Crawling & Indexing
Search Visibility Passes Through Three Stages: Crawling, Indexing and Serving
Crawling and indexing is the process in which a search engine follows links and sitemaps to discover pages and download their content (crawling), then analyzes that content and stores it in its search index (indexing). Google Search Central’s How Google Search Works document explains that search works in three stages, crawling, indexing and serving, and that each stage proceeds only after the previous one is passed. Google (official) A page that isn’t indexed hasn’t even entered the ranking competition, so all search visibility starts with these two stages.
On its official How Search Works page, Google says its search index “contains hundreds of billions of webpages and is well over 100,000,000 gigabytes in size.” Google (official) A huge index doesn’t mean every page gets stored. The same Google document states clearly that “indexing isn’t guaranteed; not every page that Google processes will be indexed.” Google (official) In other words, indexing is closer to a review than an automatic registration.
| Stage | What Google’s documentation says | If it gets stuck here |
|---|---|---|
| Crawling (collecting) | Googlebot follows links and sitemaps to visit pages and download their content | The search engine doesn’t even know the page exists |
| Indexing (storing) | The collected content is analyzed and stored in Google’s index database | The page doesn’t remain in the search index |
| Serving (showing) | Relevance to the query and quality are calculated and results are returned | The page is indexed but ranks low |
Diagnose “It’s Not Showing in Search” from Top to Bottom
When a page doesn’t appear in search, narrow down the cause by working your way down the pipeline from the top. Sticking to the order is the key.
- Check whether it’s indexed – Use the URL Inspection tool in Google Search Console to see whether the page is registered. If it already is, the issue is ranking, not crawling or indexing.
- Check for crawl blocks – Make sure robots.txt isn’t Disallowing that path.
- Check for noindex – Look at the meta robots tag in the page source and the X-Robots-Tag header in the HTTP response.
- Check the canonical – If another URL is designated as canonical, this page drops out of the index and is consolidated into the canonical URL.
- Discovery and quality issues – If the four checks above are fine, look at weak internal linking, missing sitemap entries and content quality.
The Page indexing report in Google Search Console tells you which stage a page is stuck at with statuses such as “Crawled – currently not indexed.” GSC Help Center Starting your diagnosis here instead of guessing saves a lot of time.
noindex, robots.txt Blocking and Canonicals Work at Different Stages
All three are easy to lump together as “ways to remove pages from search,” but according to Google’s documentation, each works at a different stage.
| Method | What it says | Crawling | Indexing |
|---|---|---|---|
| robots.txt Disallow | “Don’t visit this path” | Blocked | Can’t block directly – the URL can be indexed if found through external links |
| meta robots noindex | “Visit, but don’t store it” | Allowed | Blocked |
| Canonical tag | “Of these URLs, this one is the representative” | Allowed | Consolidated into the canonical URL (a hint, not a rule, per Google’s documentation) |
Google’s Introduction to robots.txt document states firmly that robots.txt “is not a mechanism for keeping a web page out of Google.” Google (official) Even a Disallowed page can have its URL indexed if other sites link to it, and to exclude it reliably, Google advises using noindex or password protection.
# 1) robots.txt - Blocks crawling (visits). It does not block indexing itself
User-agent: *
Disallow: /private/
<!-- 2) meta robots - Blocks indexing (storage). Crawling must be allowed for it to be read -->
<meta name="robots" content="noindex">
# 3) HTTP response header - Blocks indexing of non-HTML files such as PDFs
X-Robots-Tag: noindex
The most common combination mistake is adding noindex to a page that is also blocked by robots.txt. Google’s Block Search indexing document explains that “for the noindex rule to be effective, the page must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” Google (official) If you block the visit itself, the crawler can’t read the noindex instruction, and if the URL is found through external links, it may stay in the index without content. To reliably keep a page out of the index, leave crawling open and use noindex.
Per Google’s canonicalization documentation, the canonical tag is “a hint, not a rule.” Google (official) Google gathers multiple signals to choose the most complete and useful page as canonical, and it may pick a different URL from the one you specified. So canonicals are for consolidating duplicates, not for excluding pages from the index.
Does Google Really Crawl Every Page on Large Sites?
On crawl budget, Google’s official guidance and measurements from large-site operators disagree. Google says most sites don’t need to worry about it, but industry data from direct log analysis shows that as sites grow, crawling itself becomes a silent bottleneck.
| Google’s official position | Most sites don’t need to worry about crawl budget. “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they’re published,” you don’t need to read the guide, and it narrows those that need to manage it down to large sites with 1 million+ pages. Google (official) |
|---|---|
| Expert measurements | On large sites, Google doesn’t crawl about half the pages at all. Botify analyzed 413 million pages and 6 billion Googlebot request logs and found that the average enterprise site had only about 51% of its pages crawled (2018). Botify data |
The two accounts diverge because they assume different site sizes. Google’s “no need to worry” assumes an ordinary site with fewer than a few thousand pages, while Botify’s “half” figure comes from logs of large sites with hundreds of thousands to millions of pages. In practice, if you have fewer than a few thousand URLs, don’t over-tinker with crawl budget; just look after your sitemap and index coverage. For large or e-commerce sites where parameters cause URLs to explode, it’s safer to use Search Console’s Crawl stats and your server logs to confirm that important pages are actually being collected.
Common Crawling and Indexing Accidents at Deployment
Many crawling and indexing problems come not from content but from the deployment and configuration stage. Check the following before and after launch.
- Remove blocks at launch – WordPress has a “Discourage search engines from indexing this site” option to block indexing during development. If you forget to turn it off after launch, the entire site drops out of the index. Always put it on your launch checklist.
- Isolate staging servers – If development or staging servers get indexed, they become duplicate content of the live site. Block them with access authentication (a password) or noindex, as Google’s documentation advises.
- Submit an XML sitemap – According to Google’s sitemap documentation, sitemaps help crawlers find important pages and when they were updated. Google (official) An XML sitemap is especially useful for pages that are hard to discover through internal links alone.
- Check JavaScript rendering – Google’s JavaScript SEO basics document explains that crawling, rendering and indexing are separate stages, and that content missing from the rendered HTML can’t be indexed. Google (official) If the main content appears only after JS runs, check what was actually collected using the rendered HTML in Search Console’s URL Inspection.
- Prevent mass generation of low-quality URLs – If filter and sort parameters generate endless URLs, crawl resources get spread thin and collection of the pages that really matter may be delayed.
Common Mistakes
- Writing noindex instructions in robots.txt – Google officially announced that it stopped supporting noindex rules in robots.txt as of September 1, 2019. Google (official) Handle index exclusion with meta robots or X-Robots-Tag.
- Using canonicals to exclude pages from the index – Per Google’s documentation, a canonical is a non-binding hint and can be ignored. Reliable exclusion is noindex’s job.
- Mistaking indexing for a ranking guarantee – Google’s documentation explains that a separate serving (ranking) stage comes after indexing. Indexing is just the starting line.
- Getting impatient right after launch – Google says it doesn’t guarantee indexing, and crawl frequency and timing vary by site. If you’ve submitted a sitemap and done URL inspection and indexing requests, waiting is part of the diagnosis.
References
- In-Depth Guide to How Google Search Works – Google Search Central
- How Search Works: Organizing Information (index size) – Google
- robots.txt Introduction and Guide – Google Search Central
- Block Search Indexing with noindex – Google Search Central
- What is URL Canonicalization – Google Search Central
- Managing Crawl Budget for Large Sites – Google Search Central
- Understand JavaScript SEO Basics – Google Search Central
- Build and Submit a Sitemap – Google Search Central
- Page Indexing Report – Google Search Console Help
- A Note on Unsupported Rules in robots.txt (end of noindex support) – Google Search Central Blog
- Fact: Google Doesn’t Know Half Of Your Website (crawl ratio data) – Botify
- Google to Stop Supporting noindex Directive in robots.txt – Search Engine Land
- Google Cancels Support for robots.txt noindex – Search Engine Journal
Jul.2026By Studio JT