AI Crawler
What are AI crawlers, and why do they matter?
AI crawlers are bots that AI services such as ChatGPT, Claude, Perplexity and Gemini run to collect web content. Just as traditional search engines’ Googlebot visits sites for crawling and indexing, AI crawlers visit to gather model training data, build AI search indexes, and read pages in real time when users ask. OpenAI, Anthropic and Perplexity each publish the names, purposes and control methods of their crawlers in their official docs.
The scale can no longer be ignored. According to Vercel’s network analysis from late 2024, OpenAI’s GPTBot made 569 million requests in one month, and Anthropic’s Claude crawler made 370 million. SEJ (field data) Over the same period, the combined traffic of four major AI crawlers (GPTBot, Claude, AppleBot, PerplexityBot) reached about 28% of Googlebot’s request volume (4.5 billion). SEJ (field data)
Blocking means something different than in traditional search. OpenAI’s official docs say that sites that block its search bot, OAI-SearchBot, won’t be shown in ChatGPT search answers. OpenAI (official) From a GEO (Generative Engine Optimization) perspective, whether you allow crawlers is the starting point of every visibility strategy.
Training, search and user requests – telling them apart by purpose
Even the same company runs separate bots for different purposes. OpenAI’s official docs distinguish GPTBot for training, OAI-SearchBot for search indexing, and ChatGPT-User, which opens pages when a user asks. OpenAI (official) Anthropic’s Help Center also lists three types: ClaudeBot for training, Claude-SearchBot for search, and Claude-User for user requests. Anthropic (official) Perplexity’s official docs make clear that PerplexityBot is not for model training but for visibility in Perplexity search. Perplexity (official)
| Crawler | Operator | Purpose | Effect of blocking (per official docs) |
|---|---|---|---|
| GPTBot | OpenAI | Training (collecting model training data) | Signals you don’t want to provide training data. Search visibility is handled separately by OAI-SearchBot |
| OAI-SearchBot | OpenAI | Search and citation (ChatGPT search index) | Your site won’t appear in ChatGPT search answers |
| ChatGPT-User | OpenAI | Real-time visits on user request | User-initiated, so robots.txt may not apply |
| ClaudeBot | Anthropic | Training | Your future content is excluded from training datasets |
| Claude-SearchBot | Anthropic | Indexing for search quality | Excluded from the search optimization index |
| Claude-User | Anthropic | Real-time visits on user request | Pages can’t be retrieved for user questions |
| PerplexityBot | Perplexity | Search and citation (not used for model training) | Excluded from Perplexity search results |
| Perplexity-User | Perplexity | Real-time visits on user request | Generally ignores robots.txt. Block at the server level, e.g. with a WAF |
| Google-Extended | Control token (whether Gemini may use content for training and grounding) | Excluded from Gemini training and grounding. No effect on Google Search inclusion or ranking |
Google-Extended is especially easy to confuse. Google Search Central’s crawler documentation explains that Google-Extended isn’t a separate crawler but a standalone product token used in robots.txt, and that it only controls whether content Google has collected is used for Gemini training and grounding. Google (official) The same doc states that Google-Extended does not affect whether a site is included in Google Search and is not used as a ranking signal. Google (official) Google’s documentation on AI features also says that appearing in AI Overviews is handled by robots.txt rules for Googlebot and by snippet controls (nosnippet, max-snippet), and can’t be prevented by blocking Google-Extended. Google (official)
robots.txt allow/block example
You control this per User-agent in robots.txt . Whether to provide training data and whether to appear in AI answers are separate decisions, so combinations like “refuse training but allow search and citation” are possible.
# AI search and citation bots - allow (if you want to be cited in AI answers)
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# AI training bots - choose according to your site policy
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
However, you need to know two limits. First, Google Search Central’s introduction to robots.txt states that robots.txt instructions can’t enforce crawler behavior; it’s up to the crawler to obey them. Google (official) Bots that ignore the rules must be blocked separately at the server or firewall level. Second, user-request bots may not be subject to robots.txt. OpenAI’s official docs say that because ChatGPT-User’s actions are initiated by a user, robots.txt rules may not apply. OpenAI (official) Perplexity’s official docs also state that Perplexity-User generally ignores robots.txt rules. Perplexity (official)
The gap between official compliance claims and field data – the Perplexity case
Yet even bots officially said to follow robots.txt may behave differently in practice. The case that brought this issue to the fore in 2025 was Perplexity. Its official docs and third-party measurements directly contradict each other.
| Official position (Perplexity) | It advises allowing PerplexityBot in robots.txt and describes the bot as a crawler that follows robots.txt rules. Perplexity (official) |
|---|---|
| Expert field data (Cloudflare) | In August 2025, Cloudflare observed Perplexity bypassing no-crawl directives with disguised, undeclared crawlers (impersonating Chrome on macOS, rotating IPs and ASNs) across tens of thousands of domains and millions of requests a day, and on that basis removed it from its verified bot list. Cloudflare (field data) |
The reason for the gap is simple: a compliance statement in official docs declares a policy; it isn’t a technical guarantee. In practice, rather than taking official statements at face value, it’s safer to check actual access in your server logs and to block content that really must be protected firmly at the firewall or CDN level.
Practical checkpoints from a web agency’s perspective
- Serve key content as server-rendered HTML. According to Vercel’s analysis, major AI crawlers fetch JavaScript files but don’t render (execute) them. SEJ (field data) Content drawn only with JavaScript is invisible to them, so a setup that sends complete HTML from the server works in your favor.
- Match your server logs against the official IP lists. Check visits by filtering User-Agent in access.log, but verification is needed because some bots spoof their User-Agent. OpenAI publishes IP lists for each bot (gptbot.json, searchbot.json, chatgpt-user.json) in its official docs. OpenAI (official)
- Check your CDN and security settings too. Since July 2025, Cloudflare has switched to blocking AI crawlers by default for newly signed-up sites. Search Engine Land report Allowing crawlers in robots.txt means nothing if your firewall blocks them, so check that crawlers actually get a 200 response.
- As a supplementary measure, consider adopting llms.txt . It’s a specification proposed by llmstxt.org that points AI to your site’s key documents in an AI-friendly format. You can design it together with your crawler access policy.
Common mistakes
- Blocking only GPTBot and considering “AI blocking done” – OpenAI alone has several bots, and other companies’ bots still get in. OpenAI (official)
- Lumping training opt-out and citation visibility together and Disallowing all AI bots at once – OpenAI’s official docs say that blocking OAI-SearchBot means you won’t appear in ChatGPT search answers. OpenAI (official)
- Believing that blocking Google-Extended lowers your Google Search rankings, or conversely that it prevents you from appearing in AI Overviews – according to Google Search Central’s docs, neither is true. Google (official)
- Managing only robots.txt without realizing that a security plugin, firewall or CDN is automatically blocking AI crawlers – check your logs to confirm crawlers actually get normal responses. Search Engine Land report
Further reading
References
- Google’s common crawlers (including Google-Extended) – Google Search Central
- Introduction to robots.txt – Google Search Central
- AI features and your website – Google Search Central
- OpenAI web crawlers (GPTBot, OAI-SearchBot, ChatGPT-User) – OpenAI
- Does Anthropic crawl data from the web? – Claude Help Center (Anthropic)
- Perplexity crawlers (PerplexityBot, Perplexity-User) – Perplexity
- AI Crawlers Account For 28% Of Googlebot’s Traffic, Study Finds – Search Engine Journal
- Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives – Cloudflare
- Cloudflare to block AI crawlers by default with new Pay Per Crawl initiative – Search Engine Land
Jul.2026By Studio JT