Deploying llms.txt

1. Four deployment stages and a practical effort limit

Treat llms.txt deployment as four independent stages. Start with a curated file, automate its generation, add clean markdown versions of the linked pages, and then consider a full-text variant. Each stage adds value on its own, so you can stop whenever the next one would cost more than the available evidence justifies.

Three documented gaps should limit the amount of work you invest. First, no major AI vendor says that its crawler or inference pipeline consumes third-party llms.txt files. The crawler documentation from OpenAI, Anthropic, and Perplexity identifies robots.txt as the control available to site owners but does not mention llms.txt. Google is explicit: “Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them” (Google Search Central, July 2026). Second, no official validator exists. Chrome’s Lighthouse audit checks whether the file is served and treats a 404 as not applicable, so it verifies presence rather than format. Third, there is no submission channel in Search Console or Bing Webmaster Tools, nor one offered by any model provider.

The evidence supports a conservative stopping point. Ahrefs analyzed server logs across 137,210 domains in May 2026 and found that 97% of published llms.txt files received no requests at all (Ahrefs, June 2026). Two earlier measurements reached similar conclusions: a 300,000-domain study found no effect on citations, and a 90-day study of 10 sites recommended treating the file “like a sitemap: useful infrastructure, not a growth lever.”

This evidence changes the order of priorities. Demonstrated demand applies to the clean markdown in Stage 3, not to the index file in Stage 1. Cloudflare, Mintlify, and GitBook provide markdown delivery, while OpenAI, Anthropic, and Vercel publish .md versions of their own documentation. The index remains inexpensive and forward-compatible, but the markdown pages are the part that consumers already request.

Before acting on older guidance, check which version of the specification it describes. The specification was revised on 10 August 2026 and labeled v2. That revision changed the permitted file locations, reduced the significance of the Optional section, and incorporated the markdown-companion convention. The relevant changes appear below.

2. Make four decisions before deployment

Before you start, gather an inventory of pages with canonical URLs resolved, identify where the content is stored, and determine whether the pages render on the server. A markdown-based repository and a CMS database require very different amounts of work in Stage 3. You also need a list of hosts and subdomains because each file covers only the URLs below its own path. Set a token budget that you are prepared to document.

When to deploy or review it. Do this during a launch, a change to the information architecture or documentation structure, a CMS or framework migration, or the addition of a locale. The same work is appropriate when a GEO audit finds a missing or stale file, or when the published file no longer matches the site.

3. Stage 1 — curate the file

3.1 The selection rule

List the pages that help a reader answer questions about your organization or product: what it is, what it does, how to use it, where to find the reference documentation, and which policies apply. This list will differ from your highest-traffic pages. Selecting pages by traffic rather than relevance is the most common way to lose the file’s intended selectivity.

The free llms.txt Generator can create a first draft by crawling a live site’s sitemap and page metadata and organizing the result into sections. You can then apply the selection rules below by removing irrelevant pages instead of starting with an empty file.

Two structural rules follow from the specification. A consumer that truncates the file is likely to retain the blockquote, so use that line for the entity claim rather than a tagline. The H1 project name is the only required element. The Optional section also carries less weight than it did before v2. The current specification describes it as being used “by convention, for secondary information: links an agent can skip when a shorter context is needed,” and its change notes say that the section’s mechanical behavior was removed with the context-expansion tooling. Treat Optional as a hint, not as a processing contract.

Placement. v2 permits the file “at the root path /llms.txt of a website or at any subpath (e.g. /docs/llms.txt),” where a file covers the URLs below its path and the most specific applicable file takes precedence. A documentation subtree can therefore have its own file. The specification explicitly argues against /.well-known/ because well-known URIs exist only at the origin root, while many authors control only a subpath.

Multilingual sites can use one file for each locale root or a single file with separate locale sections. Choose the approach that matches the site’s hosting structure. In either case, identify the language whenever a section links to pages in a different language.

3.2 A complete example

# Northwind Analytics

> Northwind Analytics is a self-hosted product-analytics server. It ingests
> events over HTTP, stores them in ClickHouse, and exposes SQL and a query
> API. Self-hosted only; there is no managed cloud offering.

Northwind is distributed as a Docker image and a Helm chart. The docs below
are the reference material an agent should read before answering questions about
setup, querying, or limits.

## Product

- [What Northwind is](https://example.com/product.md): scope, and what it deliberately does not do
- [Pricing](https://example.com/pricing.md): per-seat and per-event tiers, with the free tier's limits

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): from Docker Compose setup to the first event in ten minutes
- [Query API](https://example.com/docs/api.md): endpoints, auth, rate limits, pagination
- [Event schema](https://example.com/docs/schema.md): required and reserved properties
- [Self-hosting](https://example.com/docs/self-hosting.md): sizing, backups, upgrade path

## Policy

- [Terms](https://example.com/terms.md): license terms for the self-hosted build
- [Security](https://example.com/security.md): disclosure process and supported versions

## Optional

- [Changelog](https://example.com/changelog.md): release history, safe to skip
- [Migration from v1](https://example.com/docs/v1-migration.md): only relevant to existing v1 operators

Three details in this example required judgment. The blockquote states a limiting fact—there is no managed cloud service—because an agent is especially likely to assume otherwise. Every link ends in .md, so Stage 3 must provide a working markdown version at each URL. The changelog appears under Optional because omitting it would not change an answer.

4. Stage 2 — generate the file from your source data

The file is useful only when it remains selective and current. On a site that changes weekly, a hand-maintained file begins going stale as soon as it is published, and no built-in warning reveals the mismatch. Generating the file solves that problem in about half a day.

Use the same data that renders the site’s index and navigation pages, such as a content collection, CMS query, or manifest. If the generator reads a different list, you have created another source that can fall out of sync.

Automation must preserve the decisions from Stage 1. Generating the file from the complete page list repeats the problem that curation was meant to solve. Add an explicit field to each page and have the generator include only pages for which that field is set.

// src/pages/llms.txt.ts — Astro builds this to /llms.txt at the site root.
// The .ts extension is stripped, so the filename carries the real extension.
import { getCollection } from 'astro:content';

const SECTIONS = ['Product', 'Docs', 'Policy', 'Optional'] as const;

export async function GET() {
  const docs = await getCollection('docs', ({ data }) =>
    // Curation survives automation: opt in per page, never "everything".
    data.inLlmsTxt && !data.draft && !data.noindex && !data.redirectFrom
  );

  const body = SECTIONS.map((section) => {
    const rows = docs
      .filter((d) => d.data.llmsSection === section)
      .sort((a, b) => (a.data.llmsOrder ?? 99) - (b.data.llmsOrder ?? 99))
      .map((d) => `- [${d.data.title}](https://example.com/${d.slug}.md): ${d.data.summary}`);
    return rows.length ? `## ${section}\n\n${rows.join('\n')}` : null;
  }).filter(Boolean).join('\n\n');

  return new Response(
    `# Northwind Analytics\n\n> ${SUMMARY}\n\n${body}\n`,
    { headers: { 'Content-Type': 'text/plain; charset=utf-8' } },
  );
}

Astro converts each src/pages/*.ts endpoint into a static file named after the source file without the .ts suffix. At build time, it calls the exported GET function and writes the response body to that file (Astro endpoints). In Next.js, the equivalent implementation is a Route Handler that returns a Response with an explicit Content-Type header (Next.js route.js).

Set the content type explicitly. A host will usually serve a static .txt file as text/plain, but a generated route uses the framework’s default unless you override it. Browsers display either response in much the same way, so the error can be easy to miss. On a static host, set the header in the platform’s configuration file. Cloudflare Pages, for example, applies path-specific rules from a plain-text _headers file.

5. Stage 3 — provide markdown versions of the linked pages

5.1 Markdown twins

The proposed benefit depends on the listed URLs returning clean text. If a link leads to an ordinary HTML page, the consumer still receives the navigation, footers, and widget markup that the convention is intended to avoid.

v2 incorporates this practice into the specification: pages “provide a clean markdown version of those pages at the same URL as the original page, either with .md appended (page.html.md) or with the extension replaced by .md (page.md).” It also defines discovery, which v1 did not address. rel="alternate" type="text/markdown" points to the markdown version of a page, while rel="describedby" points to the llms.txt file that covers it. You can provide these relations through HTML <link> elements or an HTTP Link: response header.

// src/pages/[...slug].md.ts — emits /<slug>.md next to every rendered page.
import { getCollection } from 'astro:content';

export async function getStaticPaths() {
  const docs = await getCollection('docs', ({ data }) => !data.draft);
  return docs.map((d) => ({ params: { slug: d.slug }, props: { doc: d } }));
}

export function GET({ props }) {
  const { doc } = props;
  // Title, dateline, body. No nav, no footer, no related-posts block.
  const md = `# ${doc.data.title}\n\n_Updated ${doc.data.lastUpdated}_\n\n${doc.body}`;
  return new Response(md, {
    headers: { 'Content-Type': 'text/markdown; charset=utf-8' },
  });
}

On the rendered page, advertise the markdown version so that a consumer can find it without reading llms.txt:

<link rel="alternate" type="text/markdown" href="/docs/quickstart.md">
<link rel="describedby" href="/docs/llms.txt">

The cost depends on the source format. Markdown versions are inexpensive only when the content already exists as markdown. A CMS-based site requires a conversion step, while a client-rendered site must solve server-side rendering first. A .md route built from the same empty application shell returns an empty file. SSR for AI crawlers describes this problem. Remove navigation, footers, cookie banners, related-post sections, and inline widgets. Keep the title, dateline, body, and links.

5.2 A full-text variant with a stated budget

Use an accurate name. The widely deployed llms-full.txt is a de facto convention popularized by documentation platforms; Mintlify and GitBook both generate one automatically. It does not appear in the v2 specification, and the reference tooling uses different names for its own expansion files. Choose one established convention, identify it in the file, and do not introduce another name.

The token budget is the main constraint. State a limit, measure the file during every build, and fail the build when the file exceeds it. Mintlify limits its generated llms.txt to 100,000 characters and truncates anything beyond that. No comparable limit applies to the full-text variant, which makes uncontrolled growth likely. Publish one when your pages contain reference material that an agent might read from beginning to end. Otherwise, skip it.

6. Implementation by stack

Implementation varies by stack, but the most useful distinctions are the ways each stack can fail. The table focuses on those differences.

The final row describes the same kind of failure that the AI crawler access audit checks for in robots.txt: the file in the repository may differ from the response served to users and crawlers. Only a live request shows which version is active. On WordPress, the installed plugin determines the default. Yoast is the only one of the four that applies curation automatically; the others either begin with bulk selections or rely on manually entered text. Rank Math and SEOPress serve virtual files, similar to the way WordPress core serves wp-sitemap.xml. They register a rewrite rule with add_rewrite_rule instead of writing a file to disk, which explains why there may be no file to edit.

7. Verify — seven checks

No official validator exists, so you must run these checks yourself. The first four take less than a minute.

URL="https://example.com/llms.txt"

# Content type, and the HTML-shadowing check that catches an SPA fallback.
curl -sSI "$URL" | grep -i '^content-type'
curl -sS "$URL" | head -c 32 | grep -qi '<!doctype\|<html' \
  && echo "FAIL: served HTML, not text" || echo "OK: not HTML"

# Every listed link, with its status.
curl -sS "$URL" \
  | grep -oE '\]\(https?://[^)]+\)' | tr -d '](' \
  | while read -r link; do
      printf '%-64s %s\n' "$link" "$(curl -sS -o /dev/null -w '%{http_code}' "$link")"
    done

The final comparison with the live navigation requires a person. It catches a consequential failure that automated checks miss: the file is available, every link returns 200, but the content still describes a site structure that changed two quarters ago.

8. Keep the file current or remove it

A file that no longer matches the site can mislead the consumers that choose to use it. The relevant distinction from content freshness is whether the information remains current, not whether the file carries a recent timestamp. Every line in llms.txt makes a claim about another page, so those claims must continue to match the site.

Use site changes as review triggers: a change to the information architecture, a page deletion or redirect, a product or entity rename, a restructuring of the documentation, the addition of a locale, or a migration that changes URL patterns. A quarterly spot check can catch problems that these events miss.

# Fail the build on drift rather than discovering it two quarters later.
- name: Validate llms.txt
  run: |
    set -euo pipefail
    test -s dist/llms.txt
    head -c 32 dist/llms.txt | grep -qiv '<!doctype' || { echo "served HTML"; exit 1; }
    # Roughly four characters per token; adjust to your own budget.
    chars=$(wc -c < dist/llms.txt)
    [ "$chars" -lt 100000 ] || { echo "over budget: ${chars} chars"; exit 1; }
    grep -oE '\]\(https?://[^)]+\)' dist/llms.txt | tr -d '](' \
      | xargs -I{} -P8 sh -c 'curl -sSfo /dev/null "$1" || { echo "dead: $1"; exit 1; }' _ {}

When to remove it. If no one maintains the file and you cannot generate it, deleting it is a reasonable decision. No vendor documents a penalty for an absent llms.txt file, and the largest log-based study found that 97% of existing files were never fetched. An inaccurate file, however, can mislead the consumers that choose to read it. Vendor documentation explains how to add the file, but responsible maintenance sometimes means removing it.

9. Anti-patterns

10. Deployment checklist and review triggers

• Keep the curated list to roughly 30 links or fewer, chosen for relevance rather than traffic
• Use the blockquote for the entity claim, including any important limiting fact
• Reserve Optional for links a consumer can safely skip, and treat it as a convention rather than a contract
• Serve the file on every host in scope, at the path that contains the pages it covers
• Generate it from the same source that renders the site, while honoring an explicit curation field
• Set the content type to text/plain or text/markdown and include a charset
• Confirm that every listed link resolves and that none is a redirect source
• Sample .md versions to confirm that they return markdown without surrounding interface elements (Stage 3)
• Advertise rel="alternate" and rel="describedby" from the rendered pages (Stage 3)
• Keep any full-text variant within its stated budget (Stage 4)
• Confirm that robots.txt does not disallow the path and that no WAF rule blocks it
• Require the CI checks to pass and record a named owner

Review the file after any of the changes listed in §8. Documentation from a vendor stating that it consumes third-party llms.txt files would materially change the recommendation. As of August 2026, no vendor had published such documentation; if that changes, reconsider the effort limit in §1. Review the file after any further revision of the specification as well, since the specification already changed once this month.

11. Related entries

• llms.txt — the file format and the current evidence about supply and demand
• robots.txt — the access policy that can block the file but serves a different purpose
• Sitemap and IndexNow — the discovery and completeness functions that llms.txt does not provide
• AI crawlers — the crawlers that may encounter the file
• SSR for AI crawlers — why a markdown version can be as empty as the rendered page
• Content freshness — how to judge whether the file still reflects the current site
• AI crawler access audit — how to confirm that crawlers can fetch the file
• GEO audit — how this deployment fits into a broader audit sequence