(Technical)

robots.txt

Why it matters: the first door a crawler opens

robots.txt is a plain text file placed at the site root that tells search engines and AI crawlers which paths on your site they may crawl. Google Search Central describes it as “a file that tells crawlers which URLs they can access, used mainly to avoid overloading your site with requests.” Google (official) The same document makes clear that the file must sit at the root of the target host: to control URLs under https://www.example.com/, the file must be located at https://www.example.com/robots.txt.

This approach began with the Robots Exclusion Protocol (REP) proposed by Martijn Koster in 1994, which the IETF formally standardized as RFC 9309 in September 2022. RFC 9309 The standard requires parsers to handle at least 500 KiB, and Google likewise states that it “enforces a 500 KiB (kibibyte) size limit and ignores content beyond it.” Google (official)

It has two main roles. The first is managing crawl traffic: Google’s crawl budget documentation recommends that large sites with more than one million pages use robots.txt to block URLs that are hard to consolidate, such as infinite-scroll duplicates or sort variations. Google (official) The second is serving as a policy document that states whether AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot may access the site. If your goal is to be cited in AI search, this file is the first thing to check.

Basic syntax and sample code

According to Google Search Central, rules are grouped into blocks that start with a User-agent line; disallow specifies paths you don’t want crawled, and allow specifies exception paths that override disallow. Google (official) When rules conflict, Google applies “the most specific rule based on the length of the rule path,” and when they are the same length and still conflict, it uses the least restrictive rule. Google (official)

# General search engine crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=

# Crawlers for AI search visibility - allow if you want to be cited
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Crawlers for AI model training - decide based on your content policy
User-agent: GPTBot
Allow: /

# Sitemap location as an absolute URL
Sitemap: https://www.example.com/sitemap.xml

The example blocks all of /wp-admin/ but allows admin-ajax.php as an exception because front-end features sometimes call this endpoint. Google warns, “Don’t block resources if their absence makes it harder for crawlers to understand the page.” Google (official)

The Sitemap directive

The Sitemap directive tells crawlers where your XML sitemap is. Google states that “the sitemap URL must be fully qualified, and Google doesn’t assume or check variants such as http/https or with/without www.” Google (official) The sitemap field is also supported by major search engines such as Google and Bing, and you can list as many as you like.

The biggest misconception: blocking is not removal from the index

Let’s start with the most common misconception. Google states firmly that robots.txt “is not a mechanism for keeping a web page out of Google search results.” Google (official) A blocked page can still be indexed if other sites link to it, and in that case public information such as the URL and anchor text can appear in search results. The “Indexed, though blocked by robots.txt” status in Search Console’s Page indexing report is exactly this case.

To reliably remove a page from the index, you need a noindex meta tag or an X-Robots-Tag HTTP header. Here is the crucial trap: Google states that “for the noindex rule to be effective, the page or resource must not be blocked by robots.txt, and it must be accessible to the crawler.” Google (official) If you add noindex to a page blocked by robots.txt, the crawler will never see the tag. If removal from the index is your goal, the right order is to leave robots.txt open and apply only noindex. This distinction becomes natural once you understand that crawling and indexing are separate steps.

AI crawlers and robots.txt: separate training from search

OpenAI’s official documentation distinguishes three bots. GPTBot is for training foundation models, OAI-SearchBot is the crawler that surfaces sites in ChatGPT search, and ChatGPT-User is a fetcher that visits pages in response to user questions; each bot can be controlled independently in robots.txt. OpenAI (official) Anthropic likewise lists three bots, ClaudeBot (training), Claude-SearchBot (search), and Claude-User (user requests), and states that “Anthropic’s bots respect industry-standard robots.txt directives.” Anthropic (official) Perplexity explains that PerplexityBot is for search visibility, while stating in the same document that Perplexity-User, which acts on user requests, “generally ignores robots.txt rules.” Perplexity (official)

However, official claims of “compliance” don’t always match how crawlers actually behave. This is where the vendors’ official positions and field measurements diverge.

Vendors’ official positionThe official documentation of OpenAI, Anthropic, and Perplexity states that their search crawlers respect industry-standard robots.txt directives. Perplexity (official)
Expert measurementsCloudflare’s measurements (2025) confirmed that even on new domains fully blocked with robots.txt, Perplexity got around the block with undeclared crawlers impersonating Chrome, sending 3 to 6 million requests a day, and Cloudflare removed Perplexity from its list of verified bots. Cloudflare data

The gap exists because robots.txt is a “request,” not a physical lock. Whether it is honored depends on the crawler operator’s policy, and that varies by vendor. In the same measurements, OpenAI’s ChatGPT-User was cited as a model case that stopped crawling after checking robots.txt. Cloudflare data In practice there are two paths. If your goal is to manage search and citation visibility, robots.txt is enough; but for content that absolutely must be blocked, combine it with enforceable measures such as authentication, server-side blocking, or a WAF.

Practical checkpoints from a web agency’s view

  • First check where the actual response comes from. When there is no physical file at the root, a CMS may generate and serve robots.txt dynamically. Managing both a physical file and a dynamic response leads to accidents, so open /robots.txt directly in a browser and confirm the source of the current response first.
  • Put staging blocks and their removal at launch on your deployment checklist. If a full block used during development (Disallow: /) isn’t removed when the site goes live, crawling of the entire site stays blocked.
  • Don’t block CSS and JS. Google’s documentation warns that if you block resources needed for rendering, Google can’t properly analyze the pages that depend on them. Google (official)
  • Agree on an AI crawler policy with the client. The official documentation of OpenAI, Anthropic, and Perplexity all distinguish training bots from search visibility bots, OpenAI (official) so decide separately whether to allow “providing data for model training” and “visibility in AI search,” then reflect that in the file.
  • After changes, verify with the robots.txt report in Search Console. This report shows the robots.txt files Google found, when they were last crawled, and any parsing errors or warnings, and it also lets you request a recrawl for urgent changes. GSC Help Center

Common mistakes

  • Deploying a staging “Disallow: /” to the live server as is. This is the most damaging and the most common mistake.
  • Misunderstanding path matching. According to Google’s documentation, rule paths match as prefixes, so a “/fish” rule matches every path that starts with the same string, such as /fishheads. To block only a directory, add a trailing slash. Google (official)
  • Trying to hide sensitive paths with robots.txt. Anyone can view this file at the root URL, so it actually publishes a list of the paths you want to hide. Access control must be handled with authentication.
  • Forgetting subdomains. Google states that “robots.txt rules apply only to the host, protocol, and port where the file is hosted,” so blog.example.com needs its own file. Google (official)
  • Using a relative path in the Sitemap directive. According to Google’s documentation, the sitemap URL must be a fully qualified absolute URL.

References

Jul.2026By Studio JT

More from Technical

Like this term?
Give it a like!

(63)