(Basics)

Crawling & Indexing

Search Visibility Passes Through Three Stages: Crawling, Indexing and Serving

Crawling and indexing is the process in which a search engine follows links and sitemaps to discover pages and download their content (crawling), then analyzes that content and stores it in its search index (indexing). Google Search Central’s How Google Search Works document explains that search works in three stages, crawling, indexing and serving, and that each stage proceeds only after the previous one is passed. Google (official) A page that isn’t indexed hasn’t even entered the ranking competition, so all search visibility starts with these two stages.

On its official How Search Works page, Google says its search index “contains hundreds of billions of webpages and is well over 100,000,000 gigabytes in size.” Google (official) A huge index doesn’t mean every page gets stored. The same Google document states clearly that “indexing isn’t guaranteed; not every page that Google processes will be indexed.” Google (official) In other words, indexing is closer to a review than an automatic registration.

StageWhat Google’s documentation saysIf it gets stuck here
Crawling (collecting)Googlebot follows links and sitemaps to visit pages and download their contentThe search engine doesn’t even know the page exists
Indexing (storing)The collected content is analyzed and stored in Google’s index databaseThe page doesn’t remain in the search index
Serving (showing)Relevance to the query and quality are calculated and results are returnedThe page is indexed but ranks low

Diagnose “It’s Not Showing in Search” from Top to Bottom

When a page doesn’t appear in search, narrow down the cause by working your way down the pipeline from the top. Sticking to the order is the key.

  1. Check whether it’s indexed – Use the URL Inspection tool in Google Search Console to see whether the page is registered. If it already is, the issue is ranking, not crawling or indexing.
  2. Check for crawl blocks – Make sure robots.txt isn’t Disallowing that path.
  3. Check for noindex – Look at the meta robots tag in the page source and the X-Robots-Tag header in the HTTP response.
  4. Check the canonical – If another URL is designated as canonical, this page drops out of the index and is consolidated into the canonical URL.
  5. Discovery and quality issues – If the four checks above are fine, look at weak internal linking, missing sitemap entries and content quality.

The Page indexing report in Google Search Console tells you which stage a page is stuck at with statuses such as “Crawled – currently not indexed.” GSC Help Center Starting your diagnosis here instead of guessing saves a lot of time.

noindex, robots.txt Blocking and Canonicals Work at Different Stages

All three are easy to lump together as “ways to remove pages from search,” but according to Google’s documentation, each works at a different stage.

MethodWhat it saysCrawlingIndexing
robots.txt Disallow“Don’t visit this path”BlockedCan’t block directly – the URL can be indexed if found through external links
meta robots noindex“Visit, but don’t store it”AllowedBlocked
Canonical tag“Of these URLs, this one is the representative”AllowedConsolidated into the canonical URL (a hint, not a rule, per Google’s documentation)

Google’s Introduction to robots.txt document states firmly that robots.txt “is not a mechanism for keeping a web page out of Google.” Google (official) Even a Disallowed page can have its URL indexed if other sites link to it, and to exclude it reliably, Google advises using noindex or password protection.

# 1) robots.txt - Blocks crawling (visits). It does not block indexing itself
User-agent: *
Disallow: /private/

<!-- 2) meta robots - Blocks indexing (storage). Crawling must be allowed for it to be read -->
<meta name="robots" content="noindex">

# 3) HTTP response header - Blocks indexing of non-HTML files such as PDFs
X-Robots-Tag: noindex

The most common combination mistake is adding noindex to a page that is also blocked by robots.txt. Google’s Block Search indexing document explains that “for the noindex rule to be effective, the page must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” Google (official) If you block the visit itself, the crawler can’t read the noindex instruction, and if the URL is found through external links, it may stay in the index without content. To reliably keep a page out of the index, leave crawling open and use noindex.

Per Google’s canonicalization documentation, the canonical tag is “a hint, not a rule.” Google (official) Google gathers multiple signals to choose the most complete and useful page as canonical, and it may pick a different URL from the one you specified. So canonicals are for consolidating duplicates, not for excluding pages from the index.

Does Google Really Crawl Every Page on Large Sites?

On crawl budget, Google’s official guidance and measurements from large-site operators disagree. Google says most sites don’t need to worry about it, but industry data from direct log analysis shows that as sites grow, crawling itself becomes a silent bottleneck.

Google’s official positionMost sites don’t need to worry about crawl budget. “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they’re published,” you don’t need to read the guide, and it narrows those that need to manage it down to large sites with 1 million+ pages. Google (official)
Expert measurementsOn large sites, Google doesn’t crawl about half the pages at all. Botify analyzed 413 million pages and 6 billion Googlebot request logs and found that the average enterprise site had only about 51% of its pages crawled (2018). Botify data

The two accounts diverge because they assume different site sizes. Google’s “no need to worry” assumes an ordinary site with fewer than a few thousand pages, while Botify’s “half” figure comes from logs of large sites with hundreds of thousands to millions of pages. In practice, if you have fewer than a few thousand URLs, don’t over-tinker with crawl budget; just look after your sitemap and index coverage. For large or e-commerce sites where parameters cause URLs to explode, it’s safer to use Search Console’s Crawl stats and your server logs to confirm that important pages are actually being collected.

Common Crawling and Indexing Accidents at Deployment

Many crawling and indexing problems come not from content but from the deployment and configuration stage. Check the following before and after launch.

  • Remove blocks at launch – WordPress has a “Discourage search engines from indexing this site” option to block indexing during development. If you forget to turn it off after launch, the entire site drops out of the index. Always put it on your launch checklist.
  • Isolate staging servers – If development or staging servers get indexed, they become duplicate content of the live site. Block them with access authentication (a password) or noindex, as Google’s documentation advises.
  • Submit an XML sitemap – According to Google’s sitemap documentation, sitemaps help crawlers find important pages and when they were updated. Google (official) An XML sitemap is especially useful for pages that are hard to discover through internal links alone.
  • Check JavaScript rendering – Google’s JavaScript SEO basics document explains that crawling, rendering and indexing are separate stages, and that content missing from the rendered HTML can’t be indexed. Google (official) If the main content appears only after JS runs, check what was actually collected using the rendered HTML in Search Console’s URL Inspection.
  • Prevent mass generation of low-quality URLs – If filter and sort parameters generate endless URLs, crawl resources get spread thin and collection of the pages that really matter may be delayed.

Common Mistakes

  • Writing noindex instructions in robots.txt – Google officially announced that it stopped supporting noindex rules in robots.txt as of September 1, 2019. Google (official) Handle index exclusion with meta robots or X-Robots-Tag.
  • Using canonicals to exclude pages from the index – Per Google’s documentation, a canonical is a non-binding hint and can be ignored. Reliable exclusion is noindex’s job.
  • Mistaking indexing for a ranking guarantee – Google’s documentation explains that a separate serving (ranking) stage comes after indexing. Indexing is just the starting line.
  • Getting impatient right after launch – Google says it doesn’t guarantee indexing, and crawl frequency and timing vary by site. If you’ve submitted a sitemap and done URL inspection and indexing requests, waiting is part of the diagnosis.

References

Jul.2026By Studio JT

More from Basics

Like this term?
Give it a like!

(52)