Blocked by robots.txt
What does this status mean?
‘Blocked by robots.txt’ is a status shown in the Page indexing report in Google Search Console as a reason a page is not indexed. It means a blocking rule in your site’s robots.txt file matched the URL, so Googlebot could not crawl (fetch) the page. The Google Help Center describes this status as “This page was blocked by your site’s robots.txt file.” GSC Help Center
The key point first: blocking crawling and removing a page from search results are two different things. The same document adds a warning right in the status description: “a page blocked by robots.txt can still be indexed by other means.” GSC Help Center
What robots.txt is gets its own entry. Google’s official ‘Introduction to robots.txt’ defines the file as being used “mainly to manage crawler traffic to your site” and states plainly that it “is not a mechanism for keeping a web page out of Google.” Google (official)
Is it a problem or not?
This status is not an error in itself. There is only one test: was the block intentional?
| Situation | Verdict | Action |
|---|---|---|
| You deliberately blocked URLs not needed in search, such as admin pages or internal site search results | Normal | No action needed |
| A page that should appear in search is on this list | Problem | Fix the robots.txt rule, then request recrawling |
| The goal is to keep the page out of search results completely | Wrong method | Remove the block, then apply noindex |
The most common misconception: blocked pages can still be indexed
The most common mistake is treating a robots.txt block as a way to remove pages from search results. Google’s official documentation says: “Google won’t crawl or index the content blocked by a robots.txt file, but we might still find and index a disallowed URL if it is linked from other places on the web.” Google (official)
Real-world testing shows the same result. In 2023, Patrick Stox of Ahrefs blocked two of the company’s own top-ranking blog posts with robots.txt. Both pages stayed in the search index, and their status changed to “Indexed, though blocked by robots.txt.” Rankings dropped only a position or two, but both posts lost all of their featured snippets, and Stox concluded the damage “wasn’t as bad as I thought.” Ahrefs (test data)
Search results that show only a URL with no description
Pages indexed this way look odd in search results. Google’s official documentation states: “If you use a robots.txt file to block your web page, its URL can still appear in search results, but the search result will not have a description.” Google (official) Because Google couldn’t read the page content, it can’t create a snippet (description), so the result is built only from public information such as the URL and the anchor text of links.
Search Console also has a separate warning status for this situation: “Indexed, though blocked by robots.txt.” In effect, Search Console itself shows that a page can be blocked and indexed at the same time.
Causes and how to check
There is only one cause: a blocking rule in robots.txt matches the URL path. According to Google’s official documentation, robots.txt rules are written as a combination of User-agent (the crawler the rule applies to), Disallow (a directory or page you don’t want crawled), and Allow (a directory or page that may be crawled). Google (official) For example, the file below blocks all crawlers from the /private/ path.
User-agent: *
Disallow: /private/
To see which rule was triggered, look up the URL with the URL Inspection tool in Search Console. For the full crawling and indexing flow, see the Crawling and Indexing entry.
What to do in each case
1. You blocked the URL on purpose: no action needed
If you blocked URLs that don’t need to appear in search in order to manage crawler traffic, this status is normal behavior. There’s no need to make it disappear from the report or to request indexing.
2. The page should appear in search
Find the rule in robots.txt that blocks the URL, and fix or delete it. Google’s official documentation says that once you update the file, “Google’s crawlers will automatically find and start using your robots.txt file,” and adds that if you need to refresh the cached copy faster, you can follow the procedure for submitting an updated robots.txt. Google (official) After that, requesting indexing in the URL Inspection tool can speed up the recheck.
3. The page must be kept out of search results for sure
This is the case to be most careful with. The Google Help Center explicitly says the fix for this status is to “remove the robots.txt block and use ‘noindex’.” GSC Help Center
The official documentation also gives the reason: “If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results.” Google (official) noindex only works when the crawler can read the page, so you must remove the block first for noindex to be seen.
<meta name="robots" content="noindex">
How pages excluded by noindex appear in the report is covered in the Excluded by ‘noindex’ tag entry.
robots.txt blocking vs. noindex
| Aspect | robots.txt block | noindex |
|---|---|---|
| What it blocks | Crawling (fetching the page) | Indexing (appearing in search results) |
| Guaranteed removal from the index | Not guaranteed – the page can be indexed through external links alone | Excluded from search results once the crawler reads the rule |
| Condition for it to work | The rule matches the URL path | The crawler must be able to access the page (it must not be blocked by robots.txt) |
| Where it’s applied | The robots.txt file in the site root | A meta tag on the page or an X-Robots-Tag HTTP header |
In short, this status only tells you that “Google is not allowed to read this page.” It is not a verdict of good or bad. You only need to check two things: does the block match your intent, and if your goal is to hide the page from search, should you switch to noindex?
References
- Page indexing report – Google Search Console Help
- Introduction to robots.txt – Google Search Central
- Block Search indexing with noindex – Google Search Central
- How to write and submit a robots.txt file – Google Search Central
- The Story of Blocking 2 High-Ranking Pages With Robots.txt – Ahrefs (Patrick Stox, 2023)
Jul.2026By Studio JT