{"id":254,"date":"2026-07-11T21:55:03","date_gmt":"2026-07-11T12:55:03","guid":{"rendered":"https:\/\/www.studio-jt.co.kr\/seo-glossary\/blocked-by-robots-txt\/"},"modified":"2026-11-15T18:46:51","modified_gmt":"2026-11-15T09:46:51","slug":"blocked-by-robots-txt","status":"publish","type":"seo_glossary","link":"https:\/\/www.studio-jt.co.kr\/en\/seo-glossary\/blocked-by-robots-txt\/","title":{"rendered":"Blocked by robots.txt"},"content":{"rendered":"<h2 class=\"wp-block-heading\">What does this status mean?<\/h2>\n<p class=\"wp-block-paragraph\">&#8216;Blocked by robots.txt&#8217; is a status shown in the Page indexing report in Google Search Console as a reason a page is not indexed. It means a blocking rule in your site\u2019s robots.txt file matched the URL, so Googlebot could not crawl (fetch) the page. The Google Help Center describes this status as &#8220;This page was blocked by your site&#8217;s robots.txt file.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a><\/p>\n<p class=\"wp-block-paragraph\">The key point first: blocking crawling and removing a page from search results are two different things. The same document adds a warning right in the status description: &#8220;a page blocked by robots.txt can still be indexed by other means.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a><\/p>\n<p class=\"wp-block-paragraph\">What <a href=\"\/en\/seo-glossary\/robots-txt\/\">robots.txt<\/a> is gets its own entry. Google\u2019s official &#8216;Introduction to robots.txt&#8217; defines the file as being used &#8220;mainly to manage crawler traffic to your site&#8221; and states plainly that it &#8220;is not a mechanism for keeping a web page out of Google.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n<h2 class=\"wp-block-heading\">Is it a problem or not?<\/h2>\n<p class=\"wp-block-paragraph\">This status is not an error in itself. There is only one test: was the block intentional?<\/p>\n<figure class=\"wp-block-table\"><table><thead><tr><th>Situation<\/th><th>Verdict<\/th><th>Action<\/th><\/tr><\/thead><tbody><tr><td>You deliberately blocked URLs not needed in search, such as admin pages or internal site search results<\/td><td>Normal<\/td><td>No action needed<\/td><\/tr><tr><td>A page that should appear in search is on this list<\/td><td>Problem<\/td><td>Fix the robots.txt rule, then request recrawling<\/td><\/tr><tr><td>The goal is to keep the page out of search results completely<\/td><td>Wrong method<\/td><td>Remove the block, then apply noindex<\/td><\/tr><\/tbody><\/table><\/figure>\n<h2 class=\"wp-block-heading\">The most common misconception: blocked pages can still be indexed<\/h2>\n<p class=\"wp-block-paragraph\">The most common mistake is treating a robots.txt block as a way to remove pages from search results. Google\u2019s official documentation says: &#8220;Google won&#8217;t crawl or index the content blocked by a robots.txt file, but we might still find and index a disallowed URL if it is linked from other places on the web.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n<p class=\"wp-block-paragraph\">Real-world testing shows the same result. In 2023, Patrick Stox of Ahrefs blocked two of the company\u2019s own top-ranking blog posts with robots.txt. Both pages stayed in the search index, and their status changed to &#8220;Indexed, though blocked by robots.txt.&#8221; Rankings dropped only a position or two, but both posts lost all of their featured snippets, and Stox concluded the damage &#8220;wasn\u2019t as bad as I thought.&#8221; <a class=\"jt-src jt-src--data\" href=\"https:\/\/ahrefs.com\/blog\/blocked-robots-test\/\" target=\"_blank\" rel=\"noopener\">Ahrefs (test data)<\/a><\/p>\n<h3 class=\"wp-block-heading\">Search results that show only a URL with no description<\/h3>\n<p class=\"wp-block-paragraph\">Pages indexed this way look odd in search results. Google\u2019s official documentation states: &#8220;If you use a robots.txt file to block your web page, its URL can still appear in search results, but the search result will not have a description.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> Because Google couldn\u2019t read the page content, it can\u2019t create a snippet (description), so the result is built only from public information such as the URL and the anchor text of links.<\/p>\n<p class=\"wp-block-paragraph\">Search Console also has a separate warning status for this situation: &#8220;Indexed, though blocked by robots.txt.&#8221; In effect, Search Console itself shows that a page can be blocked and indexed at the same time.<\/p>\n<h2 class=\"wp-block-heading\">Causes and how to check<\/h2>\n<p class=\"wp-block-paragraph\">There is only one cause: a blocking rule in robots.txt matches the URL path. According to Google\u2019s official documentation, robots.txt rules are written as a combination of User-agent (the crawler the rule applies to), Disallow (a directory or page you don\u2019t want crawled), and Allow (a directory or page that may be crawled). <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/create-robots-txt?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> For example, the file below blocks all crawlers from the \/private\/ path.<\/p>\n<pre class=\"wp-block-code\"><code>User-agent: *\nDisallow: \/private\/<\/code><\/pre>\n<p class=\"wp-block-paragraph\">To see which rule was triggered, look up the URL with the URL Inspection tool in Search Console. For the full crawling and indexing flow, see the <a href=\"\/en\/seo-glossary\/crawling-indexing\/\">Crawling and Indexing<\/a> entry.<\/p>\n<h2 class=\"wp-block-heading\">What to do in each case<\/h2>\n<h3 class=\"wp-block-heading\">1. You blocked the URL on purpose: no action needed<\/h3>\n<p class=\"wp-block-paragraph\">If you blocked URLs that don\u2019t need to appear in search in order to manage crawler traffic, this status is normal behavior. There\u2019s no need to make it disappear from the report or to request indexing.<\/p>\n<h3 class=\"wp-block-heading\">2. The page should appear in search<\/h3>\n<p class=\"wp-block-paragraph\">Find the rule in robots.txt that blocks the URL, and fix or delete it. Google\u2019s official documentation says that once you update the file, &#8220;Google&#8217;s crawlers will automatically find and start using your robots.txt file,&#8221; and adds that if you need to refresh the cached copy faster, you can follow the procedure for submitting an updated robots.txt. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/create-robots-txt?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> After that, requesting indexing in the URL Inspection tool can speed up the recheck.<\/p>\n<h3 class=\"wp-block-heading\">3. The page must be kept out of search results for sure<\/h3>\n<p class=\"wp-block-paragraph\">This is the case to be most careful with. The Google Help Center explicitly says the fix for this status is to &#8220;remove the robots.txt block and use &#8216;noindex&#8217;.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a><\/p>\n<p class=\"wp-block-paragraph\">The official documentation also gives the reason: &#8220;If the page is blocked by a robots.txt file or the crawler can&#8217;t access the page, the crawler will never see the noindex rule, and the page can still appear in search results.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/block-indexing?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> noindex only works when the crawler can read the page, so you must remove the block first for noindex to be seen.<\/p>\n<pre class=\"wp-block-code\"><code>&lt;meta name=\"robots\" content=\"noindex\"&gt;<\/code><\/pre>\n<p class=\"wp-block-paragraph\">How pages excluded by noindex appear in the report is covered in the <a href=\"\/en\/seo-glossary\/excluded-by-noindex\/\">Excluded by &#8216;noindex&#8217; tag<\/a> entry.<\/p>\n<h2 class=\"wp-block-heading\">robots.txt blocking vs. noindex<\/h2>\n<figure class=\"wp-block-table\"><table><thead><tr><th>Aspect<\/th><th>robots.txt block<\/th><th>noindex<\/th><\/tr><\/thead><tbody><tr><td>What it blocks<\/td><td>Crawling (fetching the page)<\/td><td>Indexing (appearing in search results)<\/td><\/tr><tr><td>Guaranteed removal from the index<\/td><td>Not guaranteed &#8211; the page can be indexed through external links alone<\/td><td>Excluded from search results once the crawler reads the rule<\/td><\/tr><tr><td>Condition for it to work<\/td><td>The rule matches the URL path<\/td><td>The crawler must be able to access the page (it must not be blocked by robots.txt)<\/td><\/tr><tr><td>Where it\u2019s applied<\/td><td>The robots.txt file in the site root<\/td><td>A meta tag on the page or an X-Robots-Tag HTTP header<\/td><\/tr><\/tbody><\/table><\/figure>\n<p class=\"wp-block-paragraph\">In short, this status only tells you that &#8220;Google is not allowed to read this page.&#8221; It is not a verdict of good or bad. You only need to check two things: does the block match your intent, and if your goal is to hide the page from search, should you switch to noindex?<\/p>\n<h2 class=\"wp-block-heading\">References<\/h2>\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203?hl=ko\" target=\"_blank\" rel=\"noopener\">Page indexing report &#8211; Google Search Console Help<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro?hl=ko\" target=\"_blank\" rel=\"noopener\">Introduction to robots.txt &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/block-indexing?hl=ko\" target=\"_blank\" rel=\"noopener\">Block Search indexing with noindex &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/create-robots-txt?hl=ko\" target=\"_blank\" rel=\"noopener\">How to write and submit a robots.txt file &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/ahrefs.com\/blog\/blocked-robots-test\/\" target=\"_blank\" rel=\"noopener\">The Story of Blocking 2 High-Ranking Pages With Robots.txt &#8211; Ahrefs (Patrick Stox, 2023)<\/a><\/li><\/ul>","protected":false},"excerpt":{"rendered":"<p>&#8216;Blocked by robots.txt&#8217; is a status meaning Google could not crawl a page because of a blocking rule in your site\u2019s robots.txt file. If you blocked the URL on purpose, it is normal and needs no action. However, Google\u2019s official documentation warns that a page blocked by robots.txt can still be indexed through other paths, and advises removing the block and using noindex if you want it kept out of search results for sure.<\/p>\n","protected":false},"template":"","glossary_categories":[17],"class_list":["post-254","seo_glossary","type-seo_glossary","status-publish","hentry","glossary_categories-search-console"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/254","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary"}],"about":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/types\/seo_glossary"}],"version-history":[{"count":1,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/254\/revisions"}],"predecessor-version":[{"id":2339,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/254\/revisions\/2339"}],"wp:attachment":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/media?parent=254"}],"wp:term":[{"taxonomy":"glossary_categories","embeddable":true,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/glossary_categories?post=254"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}