llms.txt

1. What llms.txt is

llms.txt is a single markdown file at the root of a website (/llms.txt). Jeremy Howard of Answer.AI proposed the convention in September 2024 (see the original proposal).

Definition (GEO Wiki working definition): llms.txt is a proposed publishing convention that uses one curated markdown file to tell an LLM which pages to read first and where to find clean text. It addresses a legibility problem caused by noisy HTML and finite context windows. It does not control access or provide a complete discovery file.

The proposal describes the file as “a markdown file that provides brief background information and guidance, along with links to markdown files providing more detailed information” (see Answer.AI).

Several related considerations clarify how the convention works in practice:

These distinctions place llms.txt alongside other forms of machine-readable infrastructure. Like Schema.org for AI, it can make material easier to interpret without creating a ranking or citation signal. Within Generative Engine Optimization, its role is to improve crawl-time legibility at low cost.

2. Supply-side adoption and demand-side use

The most important distinction is between supply, meaning sites that publish llms.txt, and demand, meaning AI engines that fetch and use it. Treating those two forms of adoption as equivalent creates expectations that the evidence does not support. Schema.org for AI §2 makes a similar distinction between publishing markup and gaining a ranking or citation signal.

Anthropic publishes an llms.txt file for its developer documentation, but the published documentation for ClaudeBot does not say that the crawler uses third-party files. The same is true of the documentation for GPTBot and PerplexityBot. Publishing a file is evidence of supply-side adoption, not evidence that a vendor’s crawler reads it elsewhere.

Put plainly: llms.txt today is a low-cost, forward-compatible bet on a convention — not a citation channel you can rely on. Publishing costs close to nothing. Its confirmed benefit is that a tool or system that supports the convention can read a clean, curated guide efficiently. Vendor use remains unconfirmed.

Google Search Advocate John Mueller expressed the skeptical case directly. In April 2025, he compared llms.txt to the old keywords meta tag and wrote that “none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it)” (reported by Search Engine Journal). That observation does not make the convention worthless, but it narrows the case for publishing it to the benefits described in §7.

3. The llms.txt file format

The llms.txt specification defines a short markdown document built around a project name, an optional summary and context, and curated link lists. Deploying llms.txt covers generation for specific technology stacks. The file itself follows this structure:

# Project Name

> A short blockquote summarizing the project: the key information needed
> to make sense of the rest of the file.

Zero or more free prose paragraphs (no headings) for extra context.

## Docs

- [Quickstart](https://example.com/quickstart.md): how to get started
- [API reference](https://example.com/api.md): the full endpoint list

## Optional

- [Changelog](https://example.com/changelog.md): can be skipped if the
  context window is tight

llms.txt and llms-full.txt serve different purposes. The original proposal defines processed expansions named llms-ctx.txt and llms-ctx-full.txt, which the llms_txt2ctx tool generates from llms.txt (see Answer.AI). The widely used filename llms-full.txt refers to a separate, de facto convention popularized by Mintlify: one file containing the full text of all documentation (Mintlify, 2024-11-20). In practical terms, llms.txt is a curated index, while a full variant puts the complete documentation text in a single file and may exceed an LLM’s context window.

4. How llms.txt differs from robots.txt and sitemap.xml

These three root-level files have distinct purposes. None replaces the other two.

Calling llms.txt “a sitemap for AI” confuses curation with completeness. Calling it “robots.txt for AI” confuses a reading guide with an access rule. robots.txt expresses the access decision, and the broader AI Crawlers layer determines which agents may make requests. llms.txt itself neither grants nor blocks a fetch.

5. What llms.txt does and does not do

The convention can improve legibility for tools that choose to follow it. Current evidence supports only that limited role. It does not make the underlying content more citable or create a new ranking signal, distinctions also discussed in Citability §5 and Schema.org for AI §6.

Consumption also varies by platform. As of May 2026, ChatGPT Search, Perplexity AI, and Claude did not document using third-party llms.txt files.

6. Common llms.txt mistakes

These mistakes reflect misunderstandings about the file’s purpose, the need to keep its links current, or the limits of an LLM’s context window.

The value of llms.txt depends on both curation and freshness. If the file is uncurated or out of date, it sends tools that follow the convention to the wrong material and makes the guide less reliable.

7. Why llms.txt matters for GEO

SEO vs GEO separates established technical foundations from speculative additions. llms.txt belongs in the latter category: inexpensive to publish and potentially useful, but not supported as a current citation mechanism.

The strongest case for llms.txt is efficiency within a limited context window. A clean, curated starting point reduces how much a supporting system must process, whether it is your own RAG application, an AI coding agent, or a third-party tool. Generating the file automatically costs close to nothing and prepares the site for possible future vendor adoption.

The evidence supports treating llms.txt as a legibility and retrievability aid, not as a GEO channel. The two largest studies cited here found no clear effect on citations (SEJ, 300,000 domains; Search Engine Land, 10 sites). Citability still depends on whether a retrieved passage is usable, while Answer Loop describes the stages between retrieval and a cited answer.

The debate remains open. Proponents argue that the convention has “a long road ahead, but I wouldn’t bet against it” (Search Engine Land, 2025-03-28). Skeptics note that robots.txt and sitemaps already cover much of the need. Both points can guide Generative Engine Optimization work: llms.txt can be a sensible low-cost addition without becoming a growth strategy.

8. Actions and governance

Publish llms.txt when it is inexpensive to generate and easy to keep current. Continue to express crawler-access policy in robots.txt, and improve the pages themselves if the goal is to earn citations. llms.txt guides tools that follow the convention to content; it neither grants access nor secures a citation.

References

The proposal and specification:

• Answer.AI: The /llms.txt file: a proposal to help LLMs use websites (Jeremy Howard, 2024-09-03)
• llmstxt.org: the /llms.txt file specification

Vendor crawler documentation (silent on llms.txt consumption, as of 2026-05):

• OpenAI: Overview of OpenAI Crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity: Perplexity Crawlers
• Google Search Central: Overview of Google crawlers and fetchers

Supply-side adoption and the variant convention:

• Mintlify: Simplifying docs for AI with /llms.txt (2024-11-20)

Skepticism and independent measurement:

• Search Engine Journal: Google Says LLMs.Txt Comparable To Keywords Meta Tag (John Mueller, Reddit; 2025-04-17)
• Search Engine Journal: llms.txt Shows No Clear Effect On AI Citations Based On 300K Domains (2025-11-20)
• Search Engine Land: Does llms.txt matter? A 90-day study across 10 sites (2026-01-20)

Balanced explainer:

• Search Engine Land: Meet llms.txt, a proposed standard for AI website content crawling (2025-03-28)