llms.txt
A proposal, not a standard
llms.txt is a proposal to put a Markdown summary at the root path (/llms.txt) so AI language models can quickly understand a site’s key documents. Jeremy Howard of Answer.AI published it at llmstxt.org on September 3, 2024. The starting point for understanding this term is that it is still at the proposal stage, not a standard approved by a standards body such as the W3C or IETF. llms.txt spec
Here is the background. The amount of context an LLM can read at once is too small to hold an entire site, and converting HTML mixed with navigation, ads and scripts into clean text is hard. So the idea is to hand AI a single Markdown page that says, “These are the important documents on this site.” llms.txt spec
Its role is the opposite of robots.txt. Google Search Central defines robots.txt as a file that “tells search engine crawlers which URLs the crawler can access on your site.” Google (official) While robots.txt controls access, llms.txt recommends content you would like AI to read.
Format: the llmstxt.org spec
According to the spec, the only required element is an H1 title (the site or project name). Below it goes a one-line summary in a blockquote, and under H2 sections, lists of links in the form [Name](URL): Description. llms.txt spec
# STUDIO-JT
> A web agency offering custom WordPress theme development and SEO/GEO consulting.
## Services
- [Custom theme development](https://example.com/services/theme/): How we develop WordPress themes from design mockups
- [SEO consulting](https://example.com/services/seo/): Technical SEO audit and improvement process
## Optional
- [About us](https://example.com/about/): Company history and team
The spec treats an H2 section named “Optional” specially: it holds documents that AI can skip when context is short. The same proposal also includes offering a clean Markdown version of each page at its URL with .md appended. llms.txt spec
Google’s official position and field data
Let’s start with the most important fact. Google Search Central’s “AI features and your website” document states clearly that to appear in AI Overviews or AI Mode, “you don’t need to create new machine readable files, AI text files, or markup,” and advises applying the same SEO best practices as for regular Google Search. Google (official) Real traffic data points the same way.
| Google’s official position | There is no need to create a separate “AI text file” for AI visibility, and John Mueller compared llms.txt to the keywords meta tag that search engines no longer use. Google (official) |
|---|---|
| Expert field data | In a log analysis of about 137,000 domains, 97% of valid llms.txt files received not a single request in a month, and requests from AI search bots fetching content to generate answers were about 1%. Some vendors claim more citations, but public verification is limited. Ahrefs (measured) |
The official position and field data do not clash; they point to the same conclusion. Google says llms.txt is “not needed” for search, and field data shows that, for now, there are almost no real requests. So the practical call comes down to one thing: it is cheap to publish, so you can create one, but there is no basis yet to expect a direct effect or to prioritize it over other SEO work. Where opinions actually split within Google is elsewhere, covered in the next section.
When Ahrefs surveyed the 137,210 domains in its web analytics data in 2026, about 28% had published an llms.txt, but 97% of valid files received not a single request during the one-month study period (May 2026). Ahrefs (measured) Of the requests that did come in, 96% were from bots. The largest group was SEO audit tools (21.7%), and AI search (retrieval) bots, which fetch content to actually generate answers, accounted for only about 1%. Ahrefs (measured)
Google’s John Mueller is skeptical too. He said, “no AI system has said they’re using llms.txt, and you can tell from your server logs that they don’t even check for it,” comparing it to the keywords meta tag that search engines stopped using. John Mueller (Google) He also pointed out a trust issue: if the file contains a summary different from the page content, there is no way to verify it (a potential form of cloaking).
Google differs by product: Lighthouse checks it, Search says it’s unnecessary (2026)
In May 2026, Google added a new audit category called “Agentic Browsing” to Lighthouse 13.3, and it soon appeared in PageSpeed Insights (PSI) too. One audit in this category checks for an llms.txt at the site root, and the Chrome developer documentation advises creating an llms.txt file and placing it at the site root. Chrome Developers
Yet Google’s own Search team says the opposite. Its AI features document states that you don’t need to create new machine-readable files, AI text files or markup to appear in these features, and it doesn’t even mention llms.txt. Google (official)
They seem to conflict because they look at different goals. The Search team addresses “does it affect Google Search rankings or AI Overviews?” (the answer is no), while Lighthouse asks a different question: “is this site ready for AI agents to read and work with?” It is a check of agent readiness, not a search ranking signal. On top of that, this category is still an informational item “under development,” so it does not lower your score. If the file is missing (404), the audit is marked “not applicable (N/A),” and providing the file is currently optional. Chrome Developers
In short, “Google recommends llms.txt” is only half true. If your goal is search visibility, it is unnecessary (as the Search team states). If your goal is preparing for AI agent browsing, Chrome’s developer tools optionally recommend it. That is why the industry has summed it up as “the answer depends on which Google product you ask.” Search Engine Land
Proven ways to gain AI visibility: GEO research
So what does raise visibility in AI search? The GEO paper (KDD 2024) by researchers at Princeton University and IIT Delhi tested 9 methods using 10,000 queries (GEO-bench) and a BingChat-like generative engine. The most effective methods were adding statistics, adding quotations and citing sources. The top method improved Position-Adjusted Word Count by 41% and Subjective Impression by 28% over the baseline. GEO (paper)
In particular, citing sources increased the generative engine visibility of the site ranked 5th in search results by 115.1%, while lowering the #1 site’s visibility by 30.3% on average, leveling the field in favor of lower-ranked sites. GEO (paper) Keyword stuffing, on the other hand, had no effect or backfired. Together, these methods raised generative engine visibility by up to 40%, but the effect varied by domain and query type, so a field-specific approach is needed. In short, the lever for AI visibility was not a separate file but the statistics, quotations and sources in the body text.
Practical checkpoints
- The core of AI visibility is the rendered body text. As Google’s documentation says, first check for a clear heading structure, quotable sentences and crawlability, and treat llms.txt as a secondary tool after that. Google (official)
- Control access by AI crawlers in robots.txt, not llms.txt. Google explains that whether your content is used for Gemini model training and grounding is managed with the Google-Extended token in robots.txt. Google (official)
- If you publish one, include only content that matches the page body. A different summary cannot be verified, which creates a trust issue (a potential form of cloaking).
- It is a list of links, so update it whenever your URL structure changes, and check your server access logs regularly to see whether AI crawlers actually fetch /llms.txt.
Common mistakes
- Confusing it with robots.txt: llms.txt has no blocking effect at all. The tool for controlling crawler access is robots.txt.
- Dumping every page into it: the spec’s purpose is curation, picking out the important documents, and it is designed to set priorities with an Optional section. Listing everything defeats the purpose.
- Creating it once and forgetting it: when your URL structure changes, you are left with a list of broken links.
- Thinking that publishing llms.txt completes your GEO (generative engine optimization) work: Google’s official documentation says no separate file is needed and to apply existing SEO best practices as is.
Further reading
- Setting Up ai.txt and llms.txt (Beginner’s Guide to SEO/AEO/GEO)
- The Complete Guide to AI Search Optimization (GEO): Content Strategy Backed by Research Papers
References
- AI features and your website – Google Search Central (official)
- Introduction to robots.txt – Google Search Central (official)
- List of Google’s common crawlers (Google-Extended) – Google Search Central (official)
- GEO: Generative Engine Optimization – Princeton & IIT Delhi, KDD 2024 (paper)
- The /llms.txt file – llmstxt.org (original proposal spec)
- We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read – Ahrefs (measured)
- What Is llms.txt, and Should You Care About It? – Ahrefs (explainer)
- Google Says LLMs.Txt Comparable To Keywords Meta Tag – Search Engine Journal
Jul.2026By Studio JT