(Search Console)

Blocked by robots.txt

What does this status mean?

‘Blocked by robots.txt’ is a status shown in the Page indexing report in Google Search Console as a reason a page is not indexed. It means a blocking rule in your site’s robots.txt file matched the URL, so Googlebot could not crawl (fetch) the page. The Google Help Center describes this status as “This page was blocked by your site’s robots.txt file.” GSC Help Center

The key point first: blocking crawling and removing a page from search results are two different things. The same document adds a warning right in the status description: “a page blocked by robots.txt can still be indexed by other means.” GSC Help Center

What robots.txt is gets its own entry. Google’s official ‘Introduction to robots.txt’ defines the file as being used “mainly to manage crawler traffic to your site” and states plainly that it “is not a mechanism for keeping a web page out of Google.” Google (official)

Is it a problem or not?

This status is not an error in itself. There is only one test: was the block intentional?

SituationVerdictAction
You deliberately blocked URLs not needed in search, such as admin pages or internal site search resultsNormalNo action needed
A page that should appear in search is on this listProblemFix the robots.txt rule, then request recrawling
The goal is to keep the page out of search results completelyWrong methodRemove the block, then apply noindex

The most common misconception: blocked pages can still be indexed

The most common mistake is treating a robots.txt block as a way to remove pages from search results. Google’s official documentation says: “Google won’t crawl or index the content blocked by a robots.txt file, but we might still find and index a disallowed URL if it is linked from other places on the web.” Google (official)

Real-world testing shows the same result. In 2023, Patrick Stox of Ahrefs blocked two of the company’s own top-ranking blog posts with robots.txt. Both pages stayed in the search index, and their status changed to “Indexed, though blocked by robots.txt.” Rankings dropped only a position or two, but both posts lost all of their featured snippets, and Stox concluded the damage “wasn’t as bad as I thought.” Ahrefs (test data)

Search results that show only a URL with no description

Pages indexed this way look odd in search results. Google’s official documentation states: “If you use a robots.txt file to block your web page, its URL can still appear in search results, but the search result will not have a description.” Google (official) Because Google couldn’t read the page content, it can’t create a snippet (description), so the result is built only from public information such as the URL and the anchor text of links.

Search Console also has a separate warning status for this situation: “Indexed, though blocked by robots.txt.” In effect, Search Console itself shows that a page can be blocked and indexed at the same time.

Causes and how to check

There is only one cause: a blocking rule in robots.txt matches the URL path. According to Google’s official documentation, robots.txt rules are written as a combination of User-agent (the crawler the rule applies to), Disallow (a directory or page you don’t want crawled), and Allow (a directory or page that may be crawled). Google (official) For example, the file below blocks all crawlers from the /private/ path.

User-agent: *
Disallow: /private/

To see which rule was triggered, look up the URL with the URL Inspection tool in Search Console. For the full crawling and indexing flow, see the Crawling and Indexing entry.

What to do in each case

1. You blocked the URL on purpose: no action needed

If you blocked URLs that don’t need to appear in search in order to manage crawler traffic, this status is normal behavior. There’s no need to make it disappear from the report or to request indexing.

2. The page should appear in search

Find the rule in robots.txt that blocks the URL, and fix or delete it. Google’s official documentation says that once you update the file, “Google’s crawlers will automatically find and start using your robots.txt file,” and adds that if you need to refresh the cached copy faster, you can follow the procedure for submitting an updated robots.txt. Google (official) After that, requesting indexing in the URL Inspection tool can speed up the recheck.

3. The page must be kept out of search results for sure

This is the case to be most careful with. The Google Help Center explicitly says the fix for this status is to “remove the robots.txt block and use ‘noindex’.” GSC Help Center

The official documentation also gives the reason: “If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results.” Google (official) noindex only works when the crawler can read the page, so you must remove the block first for noindex to be seen.

<meta name="robots" content="noindex">

How pages excluded by noindex appear in the report is covered in the Excluded by ‘noindex’ tag entry.

robots.txt blocking vs. noindex

Aspectrobots.txt blocknoindex
What it blocksCrawling (fetching the page)Indexing (appearing in search results)
Guaranteed removal from the indexNot guaranteed – the page can be indexed through external links aloneExcluded from search results once the crawler reads the rule
Condition for it to workThe rule matches the URL pathThe crawler must be able to access the page (it must not be blocked by robots.txt)
Where it’s appliedThe robots.txt file in the site rootA meta tag on the page or an X-Robots-Tag HTTP header

In short, this status only tells you that “Google is not allowed to read this page.” It is not a verdict of good or bad. You only need to check two things: does the block match your intent, and if your goal is to hide the page from search, should you switch to noindex?

References

Jul.2026By Studio JT

More from Search Console

Like this term?
Give it a like!

(137)