{"id":226,"date":"2026-07-11T21:55:02","date_gmt":"2026-07-11T12:55:02","guid":{"rendered":"https:\/\/www.studio-jt.co.kr\/seo-glossary\/robots-txt\/"},"modified":"2026-11-15T18:46:50","modified_gmt":"2026-11-15T09:46:50","slug":"robots-txt","status":"publish","type":"seo_glossary","link":"https:\/\/www.studio-jt.co.kr\/en\/seo-glossary\/robots-txt\/","title":{"rendered":"robots.txt"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Why it matters: the first door a crawler opens<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">robots.txt is a plain text file placed at the site root that tells search engines and AI crawlers which paths on your site they may crawl. Google Search Central describes it as &#8220;a file that tells crawlers which URLs they can access, used mainly to avoid overloading your site with requests.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> The same document makes clear that the file must sit at the root of the target host: to control URLs under https:\/\/www.example.com\/, the file must be located at https:\/\/www.example.com\/robots.txt.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This approach began with the Robots Exclusion Protocol (REP) proposed by Martijn Koster in 1994, which the IETF formally standardized as RFC 9309 in September 2022. <a class=\"jt-src jt-src--research\" href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9309.html\" target=\"_blank\" rel=\"noopener\">RFC 9309<\/a> The standard requires parsers to handle at least 500 KiB, and Google likewise states that it &#8220;enforces a 500 KiB (kibibyte) size limit and ignores content beyond it.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/robots_txt\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It has two main roles. The first is managing crawl traffic: Google&#8217;s crawl budget documentation recommends that large sites with more than one million pages use robots.txt to block URLs that are hard to consolidate, such as infinite-scroll duplicates or sort variations. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> The second is serving as a policy document that states whether <a href=\"\/en\/seo-glossary\/ai-crawler\/\">AI crawlers<\/a> such as GPTBot, ClaudeBot, and PerplexityBot may access the site. If your goal is to be cited in AI search, this file is the first thing to check.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Basic syntax and sample code<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">According to Google Search Central, rules are grouped into blocks that start with a User-agent line; disallow specifies paths you don&#8217;t want crawled, and allow specifies exception paths that override disallow. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> When rules conflict, Google applies &#8220;the most specific rule based on the length of the rule path,&#8221; and when they are the same length and still conflict, it uses the least restrictive rule. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/robots_txt\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># General search engine crawlers\nUser-agent: *\nDisallow: \/wp-admin\/\nAllow: \/wp-admin\/admin-ajax.php\nDisallow: \/?s=\n\n# Crawlers for AI search visibility - allow if you want to be cited\nUser-agent: OAI-SearchBot\nAllow: \/\n\nUser-agent: Claude-SearchBot\nAllow: \/\n\nUser-agent: PerplexityBot\nAllow: \/\n\n# Crawlers for AI model training - decide based on your content policy\nUser-agent: GPTBot\nAllow: \/\n\n# Sitemap location as an absolute URL\nSitemap: https:\/\/www.example.com\/sitemap.xml<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The example blocks all of \/wp-admin\/ but allows admin-ajax.php as an exception because front-end features sometimes call this endpoint. Google warns, &#8220;Don&#8217;t block resources if their absence makes it harder for crawlers to understand the page.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The Sitemap directive<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The Sitemap directive tells crawlers where your <a href=\"\/en\/seo-glossary\/xml-sitemap\/\">XML sitemap<\/a> is. Google states that &#8220;the sitemap URL must be fully qualified, and Google doesn&#8217;t assume or check variants such as http\/https or with\/without www.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/robots_txt\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> The sitemap field is also supported by major search engines such as Google and Bing, and you can list as many as you like.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The biggest misconception: blocking is not removal from the index<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Let&#8217;s start with the most common misconception. Google states firmly that robots.txt &#8220;is not a mechanism for keeping a web page out of Google search results.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> A blocked page can still be indexed if other sites link to it, and in that case public information such as the URL and anchor text can appear in search results. The &#8220;Indexed, though blocked by robots.txt&#8221; status in Search Console&#8217;s Page indexing report is exactly this case.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To reliably remove a page from the index, you need a noindex meta tag or an X-Robots-Tag HTTP header. Here is the crucial trap: Google states that &#8220;for the noindex rule to be effective, the page or resource must not be blocked by robots.txt, and it must be accessible to the crawler.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/block-indexing\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a> If you add noindex to a page blocked by robots.txt, the crawler will never see the tag. If removal from the index is your goal, the right order is to leave robots.txt open and apply only noindex. This distinction becomes natural once you understand that <a href=\"\/en\/seo-glossary\/crawling-indexing\/\">crawling and indexing<\/a> are separate steps.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">AI crawlers and robots.txt: separate training from search<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">OpenAI&#8217;s official documentation distinguishes three bots. GPTBot is for training foundation models, OAI-SearchBot is the crawler that surfaces sites in ChatGPT search, and ChatGPT-User is a fetcher that visits pages in response to user questions; each bot can be controlled independently in robots.txt. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.openai.com\/api\/docs\/bots\" target=\"_blank\" rel=\"noopener\">OpenAI (official)<\/a> Anthropic likewise lists three bots, ClaudeBot (training), Claude-SearchBot (search), and Claude-User (user requests), and states that &#8220;Anthropic&#8217;s bots respect industry-standard robots.txt directives.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.claude.com\/en\/articles\/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler\" target=\"_blank\" rel=\"noopener\">Anthropic (official)<\/a> Perplexity explains that PerplexityBot is for search visibility, while stating in the same document that Perplexity-User, which acts on user requests, &#8220;generally ignores robots.txt rules.&#8221; <a class=\"jt-src jt-src--official\" href=\"https:\/\/docs.perplexity.ai\/guides\/bots\" target=\"_blank\" rel=\"noopener\">Perplexity (official)<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, official claims of &#8220;compliance&#8221; don&#8217;t always match how crawlers actually behave. This is where the vendors&#8217; official positions and field measurements diverge.<\/p>\n\n\n\n<figure class=\"wp-block-table jt-verdict\"><table><tbody><tr><th>Vendors&#8217; official position<\/th><td>The official documentation of OpenAI, Anthropic, and Perplexity states that their search crawlers respect industry-standard robots.txt directives. <a class=\"jt-src jt-src--official\" href=\"https:\/\/docs.perplexity.ai\/guides\/bots\" target=\"_blank\" rel=\"noopener\">Perplexity (official)<\/a><\/td><\/tr><tr><th>Expert measurements<\/th><td>Cloudflare&#8217;s measurements (2025) confirmed that even on new domains fully blocked with robots.txt, Perplexity got around the block with undeclared crawlers impersonating Chrome, sending 3 to 6 million requests a day, and Cloudflare removed Perplexity from its list of verified bots. <a class=\"jt-src jt-src--data\" href=\"https:\/\/blog.cloudflare.com\/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives\/\" target=\"_blank\" rel=\"noopener\">Cloudflare data<\/a><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The gap exists because robots.txt is a &#8220;request,&#8221; not a physical lock. Whether it is honored depends on the crawler operator&#8217;s policy, and that varies by vendor. In the same measurements, OpenAI&#8217;s ChatGPT-User was cited as a model case that stopped crawling after checking robots.txt. <a class=\"jt-src jt-src--data\" href=\"https:\/\/blog.cloudflare.com\/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives\/\" target=\"_blank\" rel=\"noopener\">Cloudflare data<\/a> In practice there are two paths. If your goal is to manage search and citation visibility, robots.txt is enough; but for content that absolutely must be blocked, combine it with enforceable measures such as authentication, server-side blocking, or a WAF.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Practical checkpoints from a web agency&#8217;s view<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>First check where the actual response comes from.<\/strong> When there is no physical file at the root, a CMS may generate and serve robots.txt dynamically. Managing both a physical file and a dynamic response leads to accidents, so open \/robots.txt directly in a browser and confirm the source of the current response first.<\/li><li><strong>Put staging blocks and their removal at launch on your deployment checklist.<\/strong> If a full block used during development (Disallow: \/) isn&#8217;t removed when the site goes live, crawling of the entire site stays blocked.<\/li><li><strong>Don&#8217;t block CSS and JS.<\/strong> Google&#8217;s documentation warns that if you block resources needed for rendering, Google can&#8217;t properly analyze the pages that depend on them. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/li><li><strong>Agree on an AI crawler policy with the client.<\/strong> The official documentation of OpenAI, Anthropic, and Perplexity all distinguish training bots from search visibility bots, <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.openai.com\/api\/docs\/bots\" target=\"_blank\" rel=\"noopener\">OpenAI (official)<\/a> so decide separately whether to allow &#8220;providing data for model training&#8221; and &#8220;visibility in AI search,&#8221; then reflect that in the file.<\/li><li><strong>After changes, verify with the robots.txt report in Search Console.<\/strong> This report shows the robots.txt files Google found, when they were last crawled, and any parsing errors or warnings, and it also lets you request a recrawl for urgent changes. <a class=\"jt-src jt-src--official\" href=\"https:\/\/support.google.com\/webmasters\/answer\/6062598\" target=\"_blank\" rel=\"noopener\">GSC Help Center<\/a><\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Common mistakes<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>Deploying a staging &#8220;Disallow: \/&#8221; to the live server as is. This is the most damaging and the most common mistake.<\/li><li>Misunderstanding path matching. According to Google&#8217;s documentation, rule paths match as prefixes, so a &#8220;\/fish&#8221; rule matches every path that starts with the same string, such as \/fishheads. To block only a directory, add a trailing slash. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/robots_txt\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/li><li>Trying to hide sensitive paths with robots.txt. Anyone can view this file at the root URL, so it actually publishes a list of the paths you want to hide. Access control must be handled with authentication.<\/li><li>Forgetting subdomains. Google states that &#8220;robots.txt rules apply only to the host, protocol, and port where the file is hosted,&#8221; so blog.example.com needs its own file. <a class=\"jt-src jt-src--official\" href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/robots_txt\" target=\"_blank\" rel=\"noopener\">Google (official)<\/a><\/li><li>Using a relative path in the Sitemap directive. According to Google&#8217;s documentation, the sitemap URL must be a fully qualified absolute URL.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">References<\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/intro\" target=\"_blank\" rel=\"noopener\">Introduction to robots.txt &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/robots_txt\" target=\"_blank\" rel=\"noopener\">How Google interprets the robots.txt specification &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/robots\/create-robots-txt\" target=\"_blank\" rel=\"noopener\">How to write and submit a robots.txt file &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/block-indexing\" target=\"_blank\" rel=\"noopener\">Block search indexing with noindex &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/large-site-managing-crawl-budget\" target=\"_blank\" rel=\"noopener\">Crawl budget management for large sites &#8211; Google Search Central<\/a><\/li><li><a href=\"https:\/\/support.google.com\/webmasters\/answer\/6062598\" target=\"_blank\" rel=\"noopener\">robots.txt report &#8211; Google Search Console Help<\/a><\/li><li><a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc9309.html\" target=\"_blank\" rel=\"noopener\">RFC 9309: Robots Exclusion Protocol &#8211; IETF<\/a><\/li><li><a href=\"https:\/\/developers.openai.com\/api\/docs\/bots\" target=\"_blank\" rel=\"noopener\">Overview of OpenAI Crawlers &#8211; OpenAI<\/a><\/li><li><a href=\"https:\/\/support.claude.com\/en\/articles\/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler\" target=\"_blank\" rel=\"noopener\">Does Anthropic crawl data from the web? &#8211; Anthropic Support<\/a><\/li><li><a href=\"https:\/\/docs.perplexity.ai\/guides\/bots\" target=\"_blank\" rel=\"noopener\">Perplexity Crawlers &#8211; Perplexity Docs<\/a><\/li><li><a href=\"https:\/\/blog.cloudflare.com\/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives\/\" target=\"_blank\" rel=\"noopener\">Perplexity is using stealth, undeclared crawlers to evade no-crawl directives &#8211; Cloudflare<\/a><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>robots.txt is a standard text file placed at the site root that tells search engines and AI crawlers which paths on your site they may crawl. The Robots Exclusion Protocol (REP), proposed in 1994, was formally standardized by the IETF as RFC 9309 in 2022, and crawlers from OpenAI, Anthropic, and Perplexity check this file as well as Google&#8217;s. However, according to Google&#8217;s official documentation, blocking in robots.txt does not guarantee removal from the index, so it must be used separately from noindex.<\/p>\n","protected":false},"template":"","glossary_categories":[21],"class_list":["post-226","seo_glossary","type-seo_glossary","status-publish","hentry","glossary_categories-technical"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/226","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary"}],"about":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/types\/seo_glossary"}],"version-history":[{"count":1,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/226\/revisions"}],"predecessor-version":[{"id":2311,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/seo_glossary\/226\/revisions\/2311"}],"wp:attachment":[{"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/media?parent=226"}],"wp:term":[{"taxonomy":"glossary_categories","embeddable":true,"href":"https:\/\/www.studio-jt.co.kr\/en\/wp-json\/wp\/v2\/glossary_categories?post=226"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}