Reduce Crawl Rate
What it is for
Reducing crawl rate means lowering how often Googlebot visits your site to ease the load on your server. People look for it when the site slows down even though visitors haven’t surged, and the server access logs are packed with Googlebot requests.
Search Console once had a screen where you could lower this rate by sliding a bar left or right. If you came here from a search after failing to find that screen, your site settings are not wrong. The screen itself is gone. Let’s start with why you can’t find it no matter how much you dig through the Google Search Console menus.
The short answer: that setting was removed on January 8, 2024
Search Console’s crawl rate limiter tool was deprecated as of January 8, 2024. The deprecation notice was posted on the Google Search Central Blog on November 24, 2023, written by Google’s Gary Illyes and Nir Kalush. Google (official)
Industry coverage also confirmed it actually disappeared that day as announced. If you go to the old settings page address, you are now redirected to the deprecation notice. The feature had launched 15 years before its deprecation, but it was a legacy screen that never made it to the new Search Console. Search Engine Roundtable
The reason for removing it is simple. Google said that improvements to its crawling logic and other tools over time had made the tool unnecessary. Google itself wrote only that it had offered the tool for more than a decade. Google (official)
Server responses set the rate, not a slider
Today, Googlebot adjusts its own rate based on how your server responds to its requests. If a URL range keeps returning 500-range responses, it slows crawling almost immediately, and it also slows down on its own when responses get noticeably slower. Google (official)
The old tool was actually sluggish. A newly set limit could take more than a day to take effect on actual crawling. Google (official)
When it removed the tool, Google also lowered the minimum crawl rate itself to a level similar to the old floor. For sites with low search demand, this effectively respects the values site owners had set manually in the past. Google (official)
Two options for urgent situations
| Method | When to use it | How fast it takes effect | The cost |
|---|---|---|---|
| Return error status codes (500 · 503 · 429) | When you urgently need to reduce crawling for a few hours to a day or two | Fastest | Past two days, the URLs may drop out of the index |
| Googlebot report form | When your server setup can’t return error responses, or when Googlebot activity is unusual | A few days to review | Requests to increase the rate are not accepted |
Google’s official documentation advises that if you urgently need to reduce crawling, return 500, 503 or 429 instead of 200 in response to crawl requests. Google (official) Google said it kept the report form for unusual Googlebot activity and emergencies, but that the fastest way to reduce the rate is to signal it through server responses. Google (official)
The form is at search.google.com/search-console/googlebot-report and requires signing in with a Google account. Googlebot report form However, it is an exception route for cases where your server setup can’t return error responses. You need to enter an appropriate rate for your site, and review and processing can take a few days. Google (official)
Two days is the tipping point
Reducing the rate with error responses must not drag on. Google recommends not going beyond a day or two, and states that if these status codes are seen on the same URLs for several days, those URLs may be dropped from Google’s index. Google (official)
The specific timeframe is documented too. If you return 503 or 429, Googlebot retries those URLs for about two days. If it continues for more than a few days, Google may permanently slow or stop crawling your site, so return to normal responses as soon as the load subsides. Google (official)
The scope of the impact isn’t narrow either. The reduced rate applies to the entire hostname. Not only the URLs returning errors but also URLs serving content normally slow down together. Once errors decrease, the rate rises again automatically. Google (official)
Reducing crawling also has wide knock-on effects. Fewer new pages are discovered, existing pages are refreshed less often so price or stock changes show up late in search, and deleted pages stay in the index longer. In Google Ads, campaigns may be canceled or paused and ads may stop serving. Google (official)
Don’t reduce it this way
If you read “signal the rate through server responses” as “throw any error,” you’ll cause an incident. Below are one method Google explicitly says not to use, and two methods that never worked as rate controls in the first place.
4xx responses such as 401 · 403 · 404
Google’s official documentation explicitly says not to use 401 and 403 status codes to limit crawl rate. 4xx responses other than 429 do not work as a way to lower crawl rate. Google (official) Gary Illyes also said that 403 and 404 do not quickly lower the rate and may even have the opposite effect. Search Engine Roundtable
In February 2023, Google went as far as publishing a separate notice. It said more site owners and some content delivery networks (CDNs) were trying to slow Googlebot with 404 and other 4xx responses, and asked them to stop. 4xx codes other than 429 cause content to be removed from Google Search, and it is even worse when the robots.txt file itself returns a 4xx. The file is then treated as if it doesn’t exist, so Googlebot learns about the paths you had blocked. Google (official)
429 is the only exception. Google reads 429 as a signal that the server is overloaded and treats it as a server error. Google (official) Server errors (5xx) and 429 temporarily slow crawling, but URLs that are already indexed eventually drop out. If your goal is removal from the index, use another method, not a blocking response such as access forbidden (403).
The crawl-delay directive in robots.txt
Google does not support crawl-delay. The official robots.txt specification lists exactly four supported fields, user-agent, allow, disallow and sitemap, and states that crawl-delay is not supported. Google (official) Other search engines may read it, but to Googlebot it is simply a line that gets ignored.
The myth that blocking with robots.txt also removes pages from search
robots.txt is a crawl management tool to keep your site from being overwhelmed by requests, not a mechanism for keeping a page out of Google. Google (official) If other pages link to the URL with descriptive text, Google can index the URL without visiting the page. To keep it out of search results, use noindex or password protection. Google (official)
There is one more thing to watch out for. The February 2023 Google notice quoted above still lists temporarily reducing the crawl rate in Search Console as its first solution. That option disappeared on January 8, 2024. Google (official) Even Google’s own documentation has outdated passages like this, and older material tends to show up first in search. That is why people keep failing to find the screen on this topic.
Most sites don’t need to touch it
| Google’s official position | Googlebot lowers its rate on its own based on server responses and response times, and when Google removed the tool it also lowered the minimum crawl rate. Its explanation is that there is no longer any need for a separate place to adjust it. Google (official) |
|---|---|
| Expert field data | Googlebot crawling itself has increased. According to Cloudflare, it grew 96% from May 2024 to May 2025, and in April 2025 it peaked for the period at 145% above May 2024. The feeling that Googlebot suddenly increased is consistent with the trend. Cloudflare data |
Jumping straight to lowering the rate because crawling increased gets the order wrong. Google names inefficient site URL structures as the most common cause of crawl spikes. Three typical examples are faceted filters and sorting, calendars that create a URL for every date, and Dynamic Search Ads targets. Google (official)
So before touching the rate, Google strongly recommends checking with your hosting provider and opening your recent server access logs to find where the traffic is coming from. Google (official) Read alongside the crawl budget entry, it becomes clear that what needs adjusting is not Googlebot’s rate but a needlessly bloated list of URLs.
When the visitor isn’t Googlebot
First, figure out whether the requests straining your server really come from Googlebot. Check in this order.
- If the name in the logs is AdsBot, the cause is likely Dynamic Search Ads targets. This crawling repeats every 3 weeks, so if you can’t handle it, reduce your ad targets or increase server capacity. Google (official)
- The requests may be impersonating Googlebot. Google officially provides ways to verify whether a request really comes from Google, using a reverse DNS lookup or its public IP lists. Google (official)
- It could also be an AI crawler. According to Cloudflare Radar, about 30% of global web traffic is bots, and in some regions bots exceed human traffic. Cloudflare data
The AI crawler landscape flipped within a year. In share among AI crawlers only, GPTBot jumped from 5% to 30% and Meta-ExternalAgent entered at 19%, while Bytespider, formerly #1, fell from 42% to 7%. Cloudflare data If you don’t check the names in your logs, you’ll blame Googlebot and fix the wrong thing.
Blocking AI crawlers is a separate issue from search crawling. Google-Extended is just a control token used in robots.txt, not a separate crawler, and it only governs whether your content is used for Gemini training and grounding answers. It does not affect whether you appear in Google Search or your rankings. Google (official) Google-CloudVertexBot likewise only handles crawling for building Vertex AI agents that the site owner requested, and has no effect on Google Search or other products. Google (official)
References
- Deprecation notice for the Search Console crawl rate limiter tool (November 24, 2023) – Google Search Central Blog
- Reduce the Google crawl rate – Google Search Central documentation
- How HTTP status codes, and network and DNS errors affect Google Search – Google Search Central documentation
- Google’s request not to use 404s to reduce crawl rate (February 2023) – Google Search Central Blog
- How Google interprets the robots.txt specification, stating crawl-delay is unsupported – Google official documentation
- Troubleshoot Google Search crawling errors, handling overcrawling – Google Search Central documentation
- Verifying requests from Google’s crawlers and fetchers – Google Search Central documentation
- From Googlebot to GPTBot: who’s crawling your site in 2025 – Cloudflare Blog
Aug.2026By Studio JT