Crawl Budget
What is crawl budget?
Crawl budget is the amount of time and resources Google can spend crawling a site. Google’s official documentation, “Crawl budget management for large site owners,” defines it as the combination of two factors: the “crawl capacity limit” and “crawl demand.” Google (official)
The number of pages on the web is practically infinite, but the resources Googlebot can spend on crawling are limited. So Google sets limits for each site on how often and how many URLs it fetches, and if crawling, the first step of crawling and indexing, is wasted on worthless URLs, discovering and refreshing the pages that actually matter gets delayed.
Bottom line first: most sites don’t need to worry about it
To put the conclusion first, most sites with fewer than a few thousand pages don’t need to worry about crawl budget. Google’s official guide states clearly: “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they’re published, you don’t need to read this guide.” Google (official)
A 2017 Google Search Central Blog post also said that sites with fewer than a few thousand URLs are generally crawled efficiently, so most publishers don’t need to worry about crawl budget. Google (official) Google recommends crawl budget management for the following sites.
| Site type | Google’s criteria |
|---|---|
| Large sites | 1 million+ unique pages with content that changes about once a week |
| Medium or larger sites | 10,000+ unique pages with content that changes daily |
| Regardless of size | Sites with a high share of URLs classified in Search Console as “Discovered – currently not indexed” |
In other words, if a typical company site or blog with hundreds to a few thousand pages has pages missing from the index, the cause is most likely somewhere other than crawl budget. Not mistaking a non-problem for a problem is the first step to understanding this concept.
The two factors that determine crawl budget
Crawl capacity limit: as much as the server can handle
This is the maximum Google can fetch without overloading the site’s server. The official documentation calculates it from the number of simultaneous connections and the delay between fetches, explaining that the limit goes up when a site responds quickly and reliably, and that Google slows its crawling on its own when responses slow down or server errors increase. Google (official)
Crawl demand: how much Google wants to crawl
Even if the server can handle it, crawling won’t increase if Google doesn’t want it. The official documentation names three factors that drive crawl demand: “perceived inventory” (the total scale of URLs Google knows about on the site), “popularity” (more popular URLs on the internet are crawled more often) and “staleness” (recrawling to reflect content changes). Google (official)
What wastes crawl budget
The 2017 Google Search Central Blog post said that having many “low-value-add URLs” negatively affects crawling and indexing, and listed the main types. Google (official)
- Faceted navigation (parameter URLs created for every filter and sort combination) and session identifiers
- On-site duplicate content
- Soft 404 pages
- Hacked pages
- Infinite spaces (URLs generated endlessly, like a calendar’s “next month” link) and proxies
- Low-quality and spam content
The current official guide also lists infinite scrolling pages, duplicate versions that differ only in sort order, soft 404s and duplicate content as sources of crawl waste. Google (official) Google’s Gary Illyes said in 2022 that “about 60% of the internet is duplicate content,” and these duplicate URLs are considered a leading cause of eaten-up crawl budget. Ahrefs (measured) In fact, Botify, which analyzes logs from large sites, reported in 2024 that on unoptimized sites only 40% of strategic URLs are crawled each month, and the remaining 60% are not crawled regularly. Botify (measured)
How to manage crawl budget (only for sites it applies to)
If yours is a large site that meets the criteria above, here are the methods Google’s official guide recommends. Google (official)
- Clean up your URL inventory: block URLs that don’t need crawling with robots.txt, and consolidate duplicate content into a representative URL.
- Return a 404 or 410 status code for permanently removed pages to reduce unnecessary revisits.
- Keep your sitemaps up to date.
- Eliminate long redirect chains.
- Make your pages and server respond more efficiently: the faster they respond, the more pages can be crawled in the same time.
Checking how your site is crawled
You can see how Google actually crawls your site in Search Console’s Crawl Stats report (Settings → Crawl stats). Google’s Help Center documentation explains that the report shows total crawl requests, average response time, breakdowns by response code, file type and Googlebot type, and “Host status,” which shows server availability. GSC Help Center
Will more crawling boost rankings? Official position vs. measurements
Finally, let’s address one of the most common misconceptions: the idea that getting crawled more will raise your rankings. Here, Google’s official explanation and industry measurement data seem to contradict each other at first glance.
| Google’s official position | Crawl budget is not a ranking factor, and increasing the crawl rate won’t improve search rankings. Crawling is simply a prerequisite for indexing. Google (official) |
|---|---|
| What experts measured | On large sites, cleaning up crawl budget increases both indexing and traffic. In a Botify case, organic traffic doubled within three months after wasted URLs were removed. Botify (measured) |
The two aren’t contradictory; it’s a matter of scale. Crawling itself isn’t a ranking signal, but pages that aren’t crawled can’t even get to the starting line of indexing and ranking competition. If your site has a few thousand pages, forget about crawl budget and focus on content and index quality; if it has hundreds of thousands of pages or more, start by removing wasted URLs through log analysis. That’s the practical dividing line.
References
Jul.2026By Studio JT