{"id":268,"date":"2026-07-22T23:46:00","date_gmt":"2026-07-22T14:46:00","guid":{"rendered":"https:\/\/www.studio-jt.co.kr\/seo-glossary\/blocked-unauthorized-401\/"},"modified":"2026-11-15T18:46:51","modified_gmt":"2026-11-15T09:46:51","slug":"blocked-unauthorized-401","status":"publish","type":"seo_glossary","link":"https:\/\/www.studio-jt.co.kr\/en\/seo-glossary\/blocked-unauthorized-401\/","title":{"rendered":"Blocked due to unauthorized request (401)"},"content":{"rendered":"<h2 class=\"wp-block-heading\">What does this status mean?<\/h2>\n<p class=\"wp-block-paragraph\">&#8220;Blocked due to unauthorized request (401)&#8221; is a status shown as a &#8220;Not indexed&#8221; reason in Search Console&#8217;s <a href=\"\/en\/seo-glossary\/page-indexing-report\/\">Page indexing report<\/a>. Google&#8217;s official help defines it as &#8220;The page was blocked to Googlebot by a request for authorization (401 response).&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a> Simply put, the page asked visitors to &#8220;log in first,&#8221; and Googlebot could not get past that barrier.<\/p>\n<p class=\"wp-block-paragraph\">Think of a security guard at a building entrance stopping you and saying, &#8220;You need to show your employee ID to enter.&#8221; Googlebot is a visitor that neither creates nor carries an ID (login credentials), so it turns around at the door. The server is not broken and the page has not disappeared; it is closer to a normal refusal saying &#8220;identification required.&#8221;<\/p>\n<h2 class=\"wp-block-heading\">A problem, or an intentional block?<\/h2>\n<p class=\"wp-block-paragraph\">The short answer: in many cases this status is not a problem. The help documentation states that a page &#8220;is not indexed, but not necessarily because of an error,&#8221; and that &#8220;it&#8217;s fine for the URL not to be indexed if there&#8217;s a good reason.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a> There is only one test: did you intend that page to appear in search results?<\/p>\n<ul class=\"wp-block-list\"><li>No action needed &#8211; pages deliberately placed behind a login, such as admin screens, members-only content and pre-launch staging (test) sites. The 401 is proof that the lock is working as designed.<\/li><li>Action needed &#8211; when a public page that anyone should be able to see appears on this list. A firewall or security plugin is likely blocking Googlebot by mistake.<\/li><\/ul>\n<p class=\"wp-block-paragraph\">Google actually recommends this approach for private content. The official guidance says: &#8220;If you have confidential or private content that you don&#8217;t want to appear in Google Search results, the simplest and most effective way to block private URLs from appearing is to store them in a password-protected directory on your site server.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/control-what-you-share?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> Google&#8217;s John Mueller also named server-side access control, namely HTTP authentication or an IP allowlist, as the recommended way to keep staging sites out of search. <a class=\"jt-src jt-src--official\" href=\"https:\/\/johnmu.com\/staging-site-indexing\/\" target=\"_blank\" rel=\"noopener\">John Mueller&#8217;s comments<\/a> A 401 on a staging site is not an incident but a best practice.<\/p>\n<h2 class=\"wp-block-heading\">When does a page drop out of the index? Official rules for 4xx<\/h2>\n<p class=\"wp-block-paragraph\">The official documentation clearly spells out what happens to a page that keeps returning 401: &#8220;Google Search doesn&#8217;t index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index.&#8221; This rule applies equally to all 4xx codes except 429. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/http-network-errors?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> Because 401 is a 4xx code, if this status continues, the page will not be newly indexed, and pages that were already indexed will drop out of search results over time.<\/p>\n<p class=\"wp-block-paragraph\">If the page is deliberately locked, this is exactly the result you want, so leave it as is. Conversely, if a public page is returning 401, the longer you leave it, the more search traffic you lose, until it disappears from search results entirely. The same status is either normal behavior or an incident, depending on what you intended.<\/p>\n<h2 class=\"wp-block-heading\">Causes: typical cases where 401 shows up<\/h2>\n<p class=\"wp-block-paragraph\">In practice, this status usually appears in the report through one of these five routes.<\/p>\n<ul class=\"wp-block-list\"><li>A deliberately locked page became known to Google &#8211; URLs for admin screens, members-only pages or staging sites were discovered through a sitemap, internal links or external links. This is the most common case and not a problem.<\/li><li>Misidentification by a firewall or security plugin &#8211; bot protection mistakes Googlebot for a suspicious bot and demands authentication. This is a typical cause of public pages showing up with 401.<\/li><li>Access restrictions at the CDN or host &#8211; settings such as IP restrictions, country blocking or temporary protection modes return 401 to Googlebot&#8217;s requests.<\/li><li>Launching without removing a development lock &#8211; when switching the site to public, basic authentication (Basic Auth, a username and password prompt) from the staging days was left on some paths.<\/li><li>Misusing it to reduce crawl rate &#8211; deliberately returning 401 to lighten server load, which, as covered in the cautions below, does not work.<\/li><\/ul>\n<h2 class=\"wp-block-heading\">How to confirm it is really a login wall<\/h2>\n<p class=\"wp-block-paragraph\">The first check is incognito mode (a new window with no login). The help documentation also advises: &#8220;You can verify this error by visiting the page in incognito mode.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a> If opening the page in an incognito window shows a login screen or a username and password prompt, Googlebot is standing in front of the same wall.<\/p>\n<p class=\"wp-block-paragraph\">The second check is the live test in the <a href=\"\/en\/seo-glossary\/url-inspection-tool\/\">URL Inspection tool<\/a>. The help documentation explains: &#8220;This is a live test. The tool fetches and tests the URL in real time.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/9012289?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a> The 401 shown in the report is a record of past crawls, so to know accurately whether Googlebot can get in right now, use the live test.<\/p>\n<p class=\"wp-block-paragraph\">The third check is your server logs (the visit records your server keeps). First you need to determine whether the request that received the 401 was really Googlebot, because some spam bots impersonate Google. Google provides an official verification method: along with a reverse DNS lookup, &#8220;you can identify Googlebot by IP address by matching the crawler&#8217;s IP address to the lists of Google crawlers&#8217; and fetchers&#8217; IP ranges.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/verifying-googlebot?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> Ask your developer or hosting provider to &#8220;check whether the IPs of the requests that got 401 are on Google&#8217;s official list.&#8221;<\/p>\n<h2 class=\"wp-block-heading\">What to do: pages to publish vs. pages to keep locked<\/h2>\n<p class=\"wp-block-paragraph\">If the page should be public, there are only two ways to fix it. The original help text reads: &#8220;If you want Googlebot to index this page, either remove authorization requirements for this page, or else allow Googlebot to access your pages by verifying its identity.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a><\/p>\n<ul class=\"wp-block-list\"><li>Remove authorization requirements &#8211; remove the login or password requirement on the page. If basic authentication from the staging days remains, remove it.<\/li><li>Allow Googlebot access &#8211; if a firewall, security plugin or CDN is the cause, add genuine Googlebot, verified as described above, to the allowlist.<\/li><\/ul>\n<p class=\"wp-block-paragraph\">After fixing it, confirm that it passes with a live test in the URL Inspection tool and send an <a href=\"\/en\/seo-glossary\/request-indexing\/\">indexing request<\/a>. The 401 in the report is only updated after Google recrawls the page, so if the live test passes, the fix is done even if the report still looks the same.<\/p>\n<h3 class=\"wp-block-heading\">Why you should not use 401 to control crawl rate<\/h3>\n<p class=\"wp-block-paragraph\">Returning 401 because Googlebot visits strain your server does not work. Google warns: &#8220;Don&#8217;t use 401 and 403 status codes for limiting the crawl rate. The 4xx status codes, except 429, have no effect on crawl rate.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/http-network-errors?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> The crawl rate does not drop; only your indexing does. If you urgently need to reduce the rate, the official alternative is to return 500, 503 or 429 for a short period, but even this comes with a condition: &#8220;if Googlebot finds these status codes on the same URL for multiple days, the URL may be dropped from Google&#8217;s index.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/reduce-crawl-rate?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n<h3 class=\"wp-block-heading\">For pages you want locked, it&#8217;s more reliable than robots.txt<\/h3>\n<p class=\"wp-block-paragraph\">Deliberately locked pages need no changes, and switching them to a robots.txt block would be a step backward. As a sure way to keep URLs out of search results, Google advises: &#8220;If you don&#8217;t want your URL to appear in Google Search results, password-protect the files on your server, use the noindex meta tag or response header, or remove the page entirely.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro?hl=ko\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> robots.txt only blocks crawling, so a URL discovered through external links can still be indexed even without its content.<\/p>\n<p class=\"wp-block-paragraph\">John Mueller recommended server-side authentication for the same reasons. He explained that noindex only works when the page is publicly accessible, which does not suit private purposes, and that it is easy to forget to revert robots.txt when going live. As the fastest way to remove a staging site that has already been indexed, he pointed to a site removal request in Search Console. <a class=\"jt-src jt-src--official\" href=\"https:\/\/johnmu.com\/staging-site-indexing\/\" target=\"_blank\" rel=\"noopener\">John Mueller&#8217;s comments<\/a> On the exclusion period, the current Removals tool help says: &#8220;A successful request lasts about six months.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/9689846?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a><\/p>\n<h2 class=\"wp-block-heading\">How it differs from similar blocked statuses<\/h2>\n<p class=\"wp-block-paragraph\">The Page indexing report has several sibling statuses that are easy to confuse with 401. On the difference from 403 in particular, the help documentation explains: &#8220;HTTP 403 means that the user agent provided credentials, but was not granted access. However, Googlebot never provides credentials, so your server is returning this error incorrectly.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203?hl=ko\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a><\/p>\n<ul class=\"wp-block-list\"><li><a href=\"\/en\/seo-glossary\/blocked-forbidden-403\/\">Blocked due to access forbidden (403)<\/a> &#8211; If 401 is &#8220;please show your ID,&#8221; 403 is &#8220;I can&#8217;t let you in even after seeing your ID.&#8221; Googlebot never presents an ID (credentials), so returning 403 to Googlebot is usually a server configuration error.<\/li><li><a href=\"\/en\/seo-glossary\/blocked-other-4xx\/\">Blocked due to other 4xx issue<\/a> &#8211; A status grouping blocks caused by other 4xx responses (400, 429, etc.) besides 401, 403 and 404.<\/li><li><a href=\"\/en\/seo-glossary\/blocked-by-robots-txt\/\">Blocked by robots.txt<\/a> &#8211; Crawling is blocked by robots.txt. Unlike 401, it only blocks crawling and does not guarantee exclusion from the index.<\/li><li><a href=\"\/en\/seo-glossary\/not-found-404\/\">Not found (404)<\/a> &#8211; The page does not exist at all. With 401, the page exists but is locked.<\/li><li><a href=\"\/en\/seo-glossary\/server-error-5xx\/\">Server error (5xx)<\/a> &#8211; The server failed to deliver content because of a server-side failure. The difference is that with 401, the server is working normally and asking for identification.<\/li><\/ul>\n<h2 class=\"wp-block-heading\">References<\/h2>\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/support.google.com\/webmasters\/answer\/7440203?hl=ko\" target=\"_blank\" rel=\"noopener\">Page indexing report &#8211; Google Search Console Help<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/http-network-errors?hl=ko\" target=\"_blank\" rel=\"noopener\">How HTTP status codes affect Google&#8217;s crawlers &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/control-what-you-share?hl=ko\" target=\"_blank\" rel=\"noopener\">Control what you share with Google &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/reduce-crawl-rate?hl=ko\" target=\"_blank\" rel=\"noopener\">Reduce the Google crawl rate &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/verifying-googlebot?hl=ko\" target=\"_blank\" rel=\"noopener\">Verifying Googlebot &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/support.google.com\/webmasters\/answer\/9012289?hl=ko\" target=\"_blank\" rel=\"noopener\">URL Inspection tool &#8211; Google Search Console Help<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro?hl=ko\" target=\"_blank\" rel=\"noopener\">Introduction to robots.txt &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/johnmu.com\/staging-site-indexing\/\" target=\"_blank\" rel=\"noopener\">John Mueller&#8217;s post on preventing staging sites from being indexed &#8211; johnmu.com<\/a><\/li><\/ul>","protected":false},"excerpt":{"rendered":"<p>&#8220;Blocked due to unauthorized request (401)&#8221; is a status meaning Googlebot could not access a page because of an authentication requirement, such as a login, so the page was not indexed. If the page is deliberately locked, like an admin screen, members-only area or staging site, this is not an error but the normal state Google recommends. If the page should be public, remove the authentication requirement or allow Googlebot access, then confirm with a live test in the URL Inspection tool.<\/p>\n","protected":false},"template":"","glossary_categories":[17],"class_list":["post-268","seo_glossary","type-seo_glossary","status-publish","hentry","glossary_categories-search-console"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/268","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary"}],"about":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/types\/seo_glossary"}],"version-history":[{"count":1,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/268\/revisions"}],"predecessor-version":[{"id":2353,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/268\/revisions\/2353"}],"wp:attachment":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/media?parent=268"}],"wp:term":[{"taxonomy":"glossary_categories","embeddable":true,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/glossary_categories?post=268"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}