{"id":259,"date":"2026-07-11T21:55:03","date_gmt":"2026-07-11T12:55:03","guid":{"rendered":"https:\/\/www.studio-jt.co.kr\/seo-glossary\/crawl-budget\/"},"modified":"2026-11-15T18:46:51","modified_gmt":"2026-11-15T09:46:51","slug":"crawl-budget","status":"publish","type":"seo_glossary","link":"https:\/\/www.studio-jt.co.kr\/en\/seo-glossary\/crawl-budget\/","title":{"rendered":"Crawl Budget"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">What is crawl budget?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Crawl budget is the amount of time and resources Google can spend crawling a site. Google&#8217;s official documentation, &#8220;Crawl budget management for large site owners,&#8221; defines it as the combination of two factors: the &#8220;crawl capacity limit&#8221; and &#8220;crawl demand.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The number of pages on the web is practically infinite, but the resources Googlebot can spend on crawling are limited. So Google sets limits for each site on how often and how many URLs it fetches, and if crawling, the first step of <a href=\"\/en\/seo-glossary\/crawling-indexing\/\">crawling and indexing<\/a>, is wasted on worthless URLs, discovering and refreshing the pages that actually matter gets delayed.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Bottom line first: most sites don&#8217;t need to worry about it<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To put the conclusion first, most sites with fewer than a few thousand pages don&#8217;t need to worry about crawl budget. Google&#8217;s official guide states clearly: &#8220;If your site doesn&#8217;t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they&#8217;re published, you don&#8217;t need to read this guide.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A 2017 Google Search Central Blog post also said that sites with fewer than a few thousand URLs are generally crawled efficiently, so most publishers don&#8217;t need to worry about crawl budget. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/blog\/2017\/01\/what-crawl-budget-means-for-googlebot\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> Google recommends crawl budget management for the following sites.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Site type<\/th><th>Google&#8217;s criteria<\/th><\/tr><\/thead><tbody><tr><td>Large sites<\/td><td>1 million+ unique pages with content that changes about once a week<\/td><\/tr><tr><td>Medium or larger sites<\/td><td>10,000+ unique pages with content that changes daily<\/td><\/tr><tr><td>Regardless of size<\/td><td>Sites with a high share of URLs classified in Search Console as <a href=\"\/en\/seo-glossary\/discovered-currently-not-indexed\/\">&#8220;Discovered &#8211; currently not indexed&#8221;<\/a><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">In other words, if a typical company site or blog with hundreds to a few thousand pages has pages missing from the index, the cause is most likely somewhere other than crawl budget. Not mistaking a non-problem for a problem is the first step to understanding this concept.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The two factors that determine crawl budget<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Crawl capacity limit: as much as the server can handle<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">This is the maximum Google can fetch without overloading the site&#8217;s server. The official documentation calculates it from the number of simultaneous connections and the delay between fetches, explaining that the limit goes up when a site responds quickly and reliably, and that Google slows its crawling on its own when responses slow down or <a href=\"\/en\/seo-glossary\/server-error-5xx\/\">server errors<\/a> increase. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Crawl demand: how much Google wants to crawl<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Even if the server can handle it, crawling won&#8217;t increase if Google doesn&#8217;t want it. The official documentation names three factors that drive crawl demand: &#8220;perceived inventory&#8221; (the total scale of URLs Google knows about on the site), &#8220;popularity&#8221; (more popular URLs on the internet are crawled more often) and &#8220;staleness&#8221; (recrawling to reflect content changes). <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What wastes crawl budget<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The 2017 Google Search Central Blog post said that having many &#8220;low-value-add URLs&#8221; negatively affects crawling and indexing, and listed the main types. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/blog\/2017\/01\/what-crawl-budget-means-for-googlebot\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Faceted navigation (parameter URLs created for every filter and sort combination) and session identifiers<\/li><li>On-site duplicate content<\/li><li>Soft 404 pages<\/li><li>Hacked pages<\/li><li>Infinite spaces (URLs generated endlessly, like a calendar&#8217;s &#8220;next month&#8221; link) and proxies<\/li><li>Low-quality and spam content<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The current official guide also lists infinite scrolling pages, duplicate versions that differ only in sort order, soft 404s and duplicate content as sources of crawl waste. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> Google&#8217;s Gary Illyes said in 2022 that &#8220;about 60% of the internet is duplicate content,&#8221; and these duplicate URLs are considered a leading cause of eaten-up crawl budget. <a class=\"jt-src jt-src--data\" href=\"https:\/\/ahrefs.com\/blog\/crawl-budget\/\" target=\"_blank\" rel=\"noopener\">Ahrefs (measured)<\/a> In fact, Botify, which analyzes logs from large sites, reported in 2024 that on unoptimized sites only 40% of strategic URLs are crawled each month, and the remaining 60% are not crawled regularly. <a class=\"jt-src jt-src--data\" href=\"https:\/\/www.botify.com\/blog\/crawl-budget-optimization\" target=\"_blank\" rel=\"noopener\">Botify (measured)<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How to manage crawl budget (only for sites it applies to)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If yours is a large site that meets the criteria above, here are the methods Google&#8217;s official guide recommends. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Clean up your URL inventory: block URLs that don&#8217;t need crawling with robots.txt, and consolidate duplicate content into a representative URL.<\/li><li>Return a 404 or 410 status code for permanently removed pages to reduce unnecessary revisits.<\/li><li>Keep your sitemaps up to date.<\/li><li>Eliminate long redirect chains.<\/li><li>Make your pages and server respond more efficiently: the faster they respond, the more pages can be crawled in the same time.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Checking how your site is crawled<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You can see how Google actually crawls your site in Search Console&#8217;s Crawl Stats report (Settings \u2192 Crawl stats). Google&#8217;s Help Center documentation explains that the report shows total crawl requests, average response time, breakdowns by response code, file type and Googlebot type, and &#8220;Host status,&#8221; which shows server availability. <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/9679690?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Will more crawling boost rankings? Official position vs. measurements<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, let&#8217;s address one of the most common misconceptions: the idea that getting crawled more will raise your rankings. Here, Google&#8217;s official explanation and industry measurement data seem to contradict each other at first glance.<\/p>\n\n\n<figure class=\"wp-block-table jt-verdict\"><table><tbody><tr><th>Google&#8217;s official position<\/th><td>Crawl budget is not a ranking factor, and increasing the crawl rate won&#8217;t improve search rankings. Crawling is simply a prerequisite for indexing. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/blog\/2017\/01\/what-crawl-budget-means-for-googlebot\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/td><\/tr><tr><th>What experts measured<\/th><td>On large sites, cleaning up crawl budget increases both indexing and traffic. In a Botify case, organic traffic doubled within three months after wasted URLs were removed. <a class=\"jt-src jt-src--data\" href=\"https:\/\/www.botify.com\/blog\/crawl-budget-optimization\" target=\"_blank\" rel=\"noopener\">Botify (measured)<\/a><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n<p class=\"wp-block-paragraph\">The two aren&#8217;t contradictory; it&#8217;s a matter of scale. Crawling itself isn&#8217;t a ranking signal, but pages that aren&#8217;t crawled can&#8217;t even get to the starting line of indexing and ranking competition. If your site has a few thousand pages, forget about crawl budget and focus on content and index quality; if it has hundreds of thousands of pages or more, start by removing wasted URLs through log analysis. That&#8217;s the practical dividing line.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">References<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget?hl=ko\" target=\"_blank\" rel=\"noopener\">Crawl budget management for large site owners &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/blog\/2017\/01\/what-crawl-budget-means-for-googlebot\" target=\"_blank\" rel=\"noopener\">What Crawl Budget Means for Googlebot &#8211; Google Search Central Blog<\/a><\/li><li><a href=\"https:\/\/support.google.com\/webmasters\/answer\/9679690?hl=ko\" target=\"_blank\" rel=\"noopener\">Crawl Stats report &#8211; Google Search Console Help Center<\/a><\/li><li><a href=\"https:\/\/ahrefs.com\/blog\/crawl-budget\/\" target=\"_blank\" rel=\"noopener\">When Should You Worry About Crawl Budget? &#8211; Ahrefs<\/a><\/li><li><a href=\"https:\/\/www.botify.com\/blog\/crawl-budget-optimization\" target=\"_blank\" rel=\"noopener\">All About Crawl Budget Optimization &#8211; Botify<\/a><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Crawl budget is the amount of time and resources Google can spend crawling a site. It&#8217;s determined by two factors: the &#8220;crawl capacity limit,&#8221; which is how much the server can handle, and &#8220;crawl demand,&#8221; which is how much Google wants to crawl the site. Google officially says that most sites with fewer than a few thousand pages don&#8217;t need to worry about crawl budget.<\/p>\n","protected":false},"template":"","glossary_categories":[17],"class_list":["post-259","seo_glossary","type-seo_glossary","status-publish","hentry","glossary_categories-search-console"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/259","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary"}],"about":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/types\/seo_glossary"}],"version-history":[{"count":1,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/259\/revisions"}],"predecessor-version":[{"id":2344,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/259\/revisions\/2344"}],"wp:attachment":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/media?parent=259"}],"wp:term":[{"taxonomy":"glossary_categories","embeddable":true,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/glossary_categories?post=259"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}