(Dev)

A Beginner’s Guide to Setting Up ai.txt and llms.txt for AI Search Optimization (SEO/AEO/GEO)

A beginner-friendly guide that explains ai.txt and llms.txt and shows you how to write them yourself.

Why did new text files appear after generative AI?

Before AI, the model was simple: search engines visited a website, collected its pages, and in return listed them in search results and sent visitors over.

Today, many people skip the search box and ask an AI like ChatGPT directly.
The AI reads and combines content from many websites and produces an answer, so the user never needs to visit those sites.

According to Cloudflare and other sources, the share of web pages read by these AI bots grows every year, and much of it is used to train AI models or to power related services. The European Union (EU) has also started to write into law the right of copyright holders to state, “Do not use our content,” when AI trains on content (an opt-out).

In this changing environment, website owners need to answer the following questions.

  • May AI use our site’s content for study (training)?
  • If so, how far, and which types of content, will we allow?
  • When AI refers to our site, do we need to guide it on which documents to read first and how?

To answer these questions systematically, new tools called ai.txt and llms.txt have been proposed on top of the existing robots.txt, and some platforms and services have started to actually use them.


robots.txt / ai.txt / llms.txt at a glance

robots.txt

  • Location: https://yourdomain/robots.txt
  • Audience: Search engines, general crawlers, some AI bots
  • Purpose: A file that says “what may and may not be crawled”
  • Format: Plain text, rule-based
  • Standardization: A long-standing web standard (RFC-based) and the most widely used set of crawl-control rules

ai.txt

  • Location: https://yourdomain/ai.txt
  • Audience: AI crawlers, AI agents
  • Purpose: A file that explains in more detail “which content AI may train on or summarize”
  • Format: Plain text, rules + metadata (data formats, API paths, etc.)
  • Standardization: Not yet an official standard; still at the proposal/experimental stage

llms.txt

  • Location: https://yourdomain/llms.txt
  • Audience: Documents that LLMs (chatbots, etc.) refer to when generating answers
  • Purpose: A summary/index file that guides “which documents AI should read first when answering”
  • Format: Markdown text
  • Standardization: Follows the llmstxt.org unofficial standard proposed by Jeremy Howard, and spreading quickly across many tools and sites

In short:

  • robots.txt = access rules that set “who may enter and how far“
  • ai.txt = a usage notice that explains “for what purposes and to what extent AI may use our content“
  • llms.txt = a guide that collects “the key documents AI should read first when building answers“

That is a good way to think about them.


What does ai.txt do?

Put simply, ai.txt is “a user manual that tells AI how it may use your site’s content.”

More specifically, it is a file that sets out the following.

  • How much AI may read
  • For what purposes the content it reads may be used (e.g., search, summarization, Q&A answers, model training)
  • If possible, which data formats you would like AI to use first (e.g., JSON, Markdown, a dedicated API)

What can you control with ai.txt?

1) What is allowed and blocked

For example, you can split content like this.

  • Public documents, announcements, blog posts
    → AI may use them for summaries, search, and answers
  • Paid courses, internal reports, members-only boards
    → AI must not use them for training, summaries, or search

You express the scope with directives such as Allow and Disallow.

2) Data paths you want AI to check first

For AI, it is more efficient to look first at organized JSON, Markdown documents, or a search API than to read full, complex HTML. So you can add hints like these to ai.txt.

  • Sitemap: A sitemap just for documents
  • Data-Format: Preferred data format (e.g., json, markdown)
  • API-Search: Search API address

With these written down, AI can more easily understand which paths and data formats to check first.

3) Policy statements

Write your AI usage principles in a paragraph or two so that humans can understand them too.

For example:

  • “Comments and posts written by members are not used for AI training in any form.”
  • “Please use our public help articles and developer documentation only to answer questions.”

Having this in writing makes the standard much clearer when you discuss it with legal, brand, and partner teams.

Which sites need ai.txt most?

If even one of the following applies to you, it is a good idea to adopt ai.txt.

  1. Sites with lots of paid or expert content
    • Paid courses, paid news, reports, expert analysis, and so on
    • When you don’t want AI to freely train on content you charge for
  2. Community services where members write a lot
    • Online communities, message boards, review services, Q&A services, and so on
    • When you want to make clear that “member posts will not be used for training” or “only posts with consent will be used”
  3. Services with high brand, legal, or policy risk
    • Finance, healthcare, education, the public sector, and so on
    • Services that may need to keep a record of “when, and under what conditions, we allowed or blocked access”

Basic ai.txt example (beginner template)

Below is the simplest possible example.

For reference, there is no official rule that ai.txt must be written in English. However,
directives and paths such as User-Agent, Allow, Disallow, and Sitemap must be in English, and if you consider overseas AI crawlers and developers, writing the Policy statements in English as well is the sensible default.

Example assumption: The blog, docs, and help center are allowed; member and payment areas are blocked.

ai.txt for yourdomain.com
# ===========================================================
# ai.txt — General AI and Agent Access Policy
# Applies to: {{website_url}}
# Version: {{version}}
# Last-Updated: {{last_updated}}
# Maintainer: {{maintainer}}
# ===========================================================
# Purpose:
# Describe how responsible AI systems may use public content
# from {{website_url}}.
# Note: robots.txt always takes precedence for crawling rules.

User-Agent: *
Allow: /blog/
Allow: /docs/
Allow: /help/
Disallow: /members/
Disallow: /billing/
Disallow: /admin/
Disallow: /user-data/

Sitemap: {{website_url}}/sitemap.xml

# Policy:
# - Public blog, docs, and help center pages MAY be used for
#   indexing, search, summarization, and Q&A answering.
# - Member-only pages, payment/checkout flows, and any user-specific
#   or personal data MUST NOT be used for model training or use in
#   AI-generated answers.
# - Reasonable attribution to {{company_name_en}} is recommended
#   when content is shown to end users.

Here is what each item means.

  • User-agent: *
    → Applies to all AI systems and crawlers
  • Allow / Disallow
    → Separates paths AI may use from paths it must not use.
    → In the example, only /blog/, /docs/, and /help/ are allowed, and paths related to members, payments, admin, and personal data are blocked.
  • Sitemap
    → A request to “please look at this data and these documents first”
  • Policy
    → States in human-readable sentences which uses are allowed and which are not

Checklist for writing ai.txt

  1. Did you open only the areas that are truly OK to use with Allow?
    • It is safest to put member data, payment data, and internal admin pages under Disallow by default.
  2. Did you write your allow/block rules as sentences?
    • In Korean or English, state clearly “what is allowed and what is prohibited.”
  3. Did you record the last modified date?
    • Recording Last-Updated in YYYY-MM-DD format makes it easier to manage policy history and check changes later.
  4. Did you open it at the real URL?
    • Type https://yourdomain/ai.txt into a browser and check that it displays without broken encoding or typos.
  5. Does it contradict robots.txt?
    • Check that paths disallowed in robots.txt are treated consistently in ai.txt as “not usable” or “private.”
    • For example, if robots.txt says Disallow: /members/ but ai.txt allows /members/ or says it may be used for training, the two files send different messages and cause confusion.

What does llms.txt do?

llms.txt is a summary/index file that helps large language models (LLMs) like ChatGPT know “which documents to read first to understand this site.”

  1. Explains the site at a glance
    • Summarizes in 2–3 lines “what this site does” and “which topics or services it is strong in.”
  2. Suggests priority documents
    • Provides links and short descriptions along the lines of “read these documents first“: help center, feature descriptions, pricing, FAQ, terms, key service pages, and so on.
  3. Provides an organized link structure
    • Splits content into sections such as ## Docs, ## API, and ## Policies, and lists the main documents under each, so that humans can follow it easily too

The location is fixed at https://yoursite/llms.txt, and the format follows a standard Markdown document.

Why use llms.txt?

The following types of sites benefit most from llms.txt.

  1. Service websites with lots of docs or help center content
    • It is a good way to send the message “please answer questions about our service based on our official documentation.”
  2. Sites with lots of blog and SEO content
    • Organizing key posts and categories for LLMs in advance sends a signal to “refer to these posts first.”
  3. Industries where expertise and trust matter
    • In law, healthcare, finance, public services, technical support, and similar fields, it helps to gather “links to accurate policy, terms, and guide documents” to reduce wrong summaries and misuse.
  4. Content hubs covering many categories or languages
    • For book/movie/product reviews, educational content, tech blogs, and so on, organizing representative posts by category in llms.txt makes it easier for LLMs to grasp the context and generate answers.

In a word, if you are “a site that wants to introduce itself properly to AI,” llms.txt is a tool worth trying with little effort.

Basic llms.txt example (beginner template)

llms.txt is an unofficial standard, but it is best to follow the format proposed at llmstxt.org as closely as possible.

  • A # title starting with # is required
  • A 2–3 line short summary (blockquote) starting with >
  • Regular paragraphs describing the site or service
  • Under sections starting with ##,
    a link list in the form – [Document name](URL): description

The example below makes the flow clearer.

# {{site_name}}

> {{site_name}} is a {{short_description}}.
> This file helps large language models (LLMs) find the most important
> resources on {{site_name}} and answer questions accurately.

Generated for language model processing.
Last-Updated: {{YYYY-MM-DD}}

Important notes:

- This llms.txt file is a non-standard, best-effort guide for LLMs.
- It does not control crawling or access. robots.txt and server settings
  always take precedence.
- Only public, non-sensitive URLs should be listed here.
- Keep links and descriptions concise, and update this file when key
  docs or URLs change.

## Sitemaps

- [XML Sitemap]({{base_url}}/sitemap_index.xml): Includes all crawlable and indexable pages.
- [Main Sections]({{base_url}}/sitemap.xml): Core pages for users and search engines.

## Docs

- [Getting Started]({{base_url}}/docs/getting-started/): Overview of the service and first steps for new users.
- [Features]({{base_url}}/docs/features/): Key features and how they work in practice.
- [FAQ]({{base_url}}/docs/faq/): Frequently asked questions and short answers.

## Blog

- [All Articles]({{base_url}}/blog/): Main blog index covering {{main_topics}}.
- [Key Article 1]({{base_url}}/blog/{{slug_1}}/): {{one_sentence_summary_1}}
- [Key Article 2]({{base_url}}/blog/{{slug_2}}/): {{one_sentence_summary_2}}

## Services

- [Main Service]({{base_url}}/services/): Overview of core services and who they are for.

## Policies

- [Terms of Service]({{base_url}}/terms/): Contractual terms for using {{site_name}}.
- [Privacy Policy]({{base_url}}/privacy/): How user data is collected, stored, and processed.
- [Cookie Policy]({{base_url}}/cookies/): Information about cookies and tracking technologies.

## Optional

- [About]({{base_url}}/about/): Background, mission, and team information.
- [Contact]({{base_url}}/contact/): How to reach the {{site_name}} team.
- [Changelog]({{base_url}}/changelog/): Notable updates and release history.

Written in this format, the file is easy for humans to read, and AI can easily see “which documents matter and in what order to refer to them.”

Checklist for writing llms.txt

Checking just the items below covers the basics.

  • Is there an H1 title?: Write it on the very first line as # Service/site name
  • Is there a short summary in blockquote format?: A 2–3 line summary starting with >
    – Write it so readers understand “what this site does” at a glance
  • Are key document links organized as a list?: Use the [Document name](URL): description format
    – Include at least a getting-started guide, feature descriptions, FAQ, terms, privacy policy, and API docs
  • Do the URLs in llms.txt actually open?: Click each link and check for 404s, login walls, or deleted documents

How to apply this to your site (step-by-step tutorial)

Finally, here is the order to follow when you actually apply ai.txt and llms.txt.

Step 1: Define your AI usage principles first

  1. How much of your site’s content may AI use?
    • e.g., blog, help center, public announcements, public API docs
  2. What content must AI not use?
    • e.g., paid courses, paid reports, members-only boards, personal data, payment data

Once these two are settled, what to put into ai.txt and llms.txt follows naturally.

Step 2: Draft and upload ai.txt

  1. Open a text editor and create an ai.txt file.
  2. Copy the basic template above and adapt it to your site’s paths and policies.
  3. Save the file with UTF-8 encoding.
  4. Upload it to the site root (web root folder).
  5. Visit https://yourdomain/ai.txt in a browser and check that the content displays correctly.

Step 3: Draft and upload llms.txt

  • First pick 5–10 “documents you really want AI to refer to.”
    – Getting-started guide, feature descriptions, pricing, FAQ, terms of service, privacy policy, API docs, etc.
  • In Markdown format, write:
    • An H1 title
    • A blockquote summary
    • A paragraph describing the service
    • Sections (## Docs, ## API, ## Policies, etc.) and link lists.
  • Save the file as llms.txt.
  • Upload it to the site root, then open https://yourdomain/llms.txt to check it.

Step 4: Make a habit of updating them when policies change

  • Whenever your pricing plans, terms of service, or privacy policy change, update ai.txt and llms.txt as well, not just the related documents.
  • This reduces the confusion of “the documents have changed, but the AI guide files are still out of date.”

Final checklist

  • Have you written down in sentences which content on your site AI may and may not use?
  • Have you written ai.txt, stating allowed/blocked paths and policy statements?
  • Does https://yourdomain/ai.txt display properly without broken encoding?
  • Have you organized the key documents AI should refer to first (getting-started guide, FAQ, terms, etc.)?
  • Have you written llms.txt in the llmstxt.org format? (H1, summary, sections, link lists)
  • Have you visited https://yourdomain/llms.txt and confirmed the links and content are correct?
  • Have you set up an internal process so that ai.txt and llms.txt are updated whenever terms, policies, or documents change?

If you use this guide as a baseline and adjust the details to fit your site, you will have not only a “AI-friendly + content-protecting” basic setup for the generative AI era, but also an essential foundation for SEO and AEO (Answer Engine Optimization).

Related terms (SEO/GEO Glossary)

For detailed definitions of the terms used in this article, see the STUDIO-JT SEO/GEO Glossary.

Nov.2025By Studio JT

More from Dev

Like this story?
Give it a like!

(113)