robots.txt

1. What robots.txt is

robots.txt is a plain-text file at the host root (/robots.txt) declaring which paths a crawler is asked not to fetch. The Robots Exclusion Protocol behind it is older than the modern web — Martijn Koster proposed it in 1994 — and ran as a de-facto convention for ~25 years before the IETF standardized it as RFC 9309 in September 2022.

Definition (GEO Wiki working definition): robots.txt is a UTF-8 plain-text file at /robots.txt that addresses crawlers by user-agent and asks them — non-bindingly — to skip certain paths. It is the standardized expression of crawler-access intent, not access control.

The clarifier RFC 9309 surfaces of itself, before this entry says anything about AI: the rules “are not a form of access authorization,” and the protocol “is not a substitute for valid content security measures” (RFC 9309 §1). The gap between declared intent and actual enforcement is the thread §5 picks up.

What changed for GEO is not the file. It is the bot population the file addresses. Before AI crawlers, the policy decision was effectively “block Googlebot, yes or no” — one decision, one bot. After AI crawlers, the same file addresses roughly 30 declared agents split into three categories with opposite access consequences (training, retrieval, user-triggered; the categorical model lives in AI Crawlers §2). Same protocol, new audience, new mistakes.

2. How the protocol works

A robots.txt file is composed of groups — sets of rules bound to one or more user-agents — served at the host root, with a small fixed grammar for path matching. The mechanic is short enough to state in one section, and the rest of this page assumes it.

The file. UTF-8 plain text, host-scoped, served at the literal path /robots.txt with a 2xx response. A 4xx response is treated as “no rules declared” — the crawler is free to fetch any path (RFC 9309 §2.3.1.3). A 5xx is conservative: crawlers retry or treat the site as fully disallowed during the outage, depending on operator policy.

Records. A group consists of one or more User-agent: lines followed by one or more Allow: and Disallow: rules. Blank lines separate groups. Comments start with #.

Group selection is the load-bearing rule writers most often miss: a crawler picks the single most specific group matching its product token (RFC 9309 §2.2.1). Rules from non-matching groups do not merge into it. An * group never applies to a bot that has a named group elsewhere in the file.

Path matching is longest-match-wins. When Allow and Disallow rules of equal path length disagree, Google’s documented tiebreak is “the least restrictive rule wins” — i.e. Allow outranks Disallow on equal-length matches (How Google Interprets the robots.txt Specification). RFC 9309 leaves the tiebreak to operator choice; most major crawlers follow Google’s convention.

Wildcards. * matches zero or more characters; $ anchors to the end of the URL. Both are supported by Google, Bing, OpenAI, Anthropic, and Perplexity, evaluated per rule rather than as one combined regex.

Sitemap directive. Sitemap: https://... is hostless — it applies file-wide and is not bound to any group. Sitemap mechanics belong in Sitemap & IndexNow.

The worked precedence example:

User-agent: GPTBot
Disallow: /
Allow: /blog/

User-agent: *
Disallow: /private/

Reading: GPTBot picks the GPTBot group, sees Disallow: / plus Allow: /blog/, and concludes “fetch nothing except /blog/.” Every other bot picks the * group and only loses /private/. The * rule never reaches GPTBot, even though it looks like a sensible default — GPTBot has its own group and consumes only that group. The same logic powers §3’s recipes and §6’s headline anti-pattern.

3. The AI-bot picture, expressed in robots.txt syntax

The categorical model (training / retrieval / user-triggered) translates into robots.txt as named user-agent groups, one cluster per category. The categorical model itself — why the splits matter — sits in AI Crawlers §2; the recipes below are how you write the file once that policy is decided.

Opt out of training only — keep AI citation, drop future parametric memory:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

Retrieval and user-triggered fetchers stay untouched, so live ChatGPT, Claude, Perplexity, and Google AI Overviews answers can still cite you.

Opt out of retrieval — accept that you will not appear in AI answers now:

User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Bingbot
Disallow: /

One caveat the protocol cannot express: blocking Googlebot removes you from Google Search and AI Overviews together — Google does not publish a separate “Search only, no AI Overviews” token. Google-Extended governs training use of fetched content, not crawling, and is therefore in the training-only block above, not this one.

Allow everything, named explicitly:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

This is identical in effect to an empty robots.txt, but stating it explicitly serves two purposes: it documents intent for internal review and audit tools, and it survives the unfortunate case where a downstream config layer prepends a blanket Disallow: / for *.

The merge gotcha, stated plainly. Because each named bot picks only its own group, the most common writer’s bug is a blanket Disallow: / under * combined with per-bot Allow: lines in named groups. It silently fails — see §6 for the canonical broken pattern.

The product-token list each vendor publishes — and the IP ranges and verification recipes — lives in the per-bot entries: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, OAI-SearchBot, ChatGPT-User. The fresh-as-of-now, audit-grade version of the recipe — what to grep in logs, how to verify what actually reached you — is the AI Crawler Access Audit playbook’s job.

4. What each major operator documents about robots.txt compliance

Documented compliance posture varies cleanly by operator. The table summarizes what each major AI-crawler operator publishes about how its bots handle robots.txt — and the carve-out each one makes for user-initiated fetches.

The user-triggered footnote, recurring as a documented operator pattern rather than a bug. ChatGPT-User, Claude-User, and Perplexity-User are each described by their operators as user-initiated and generally not subject to standard robots.txt restrictions. The reasoning each operator offers is identical: a fetch driven by one human asking about one URL is a proxy for human browsing, not autonomous crawling. OpenAI surfaced the framing most explicitly in the late-2025 docs revision; Perplexity’s help page has carried the same framing since 2024. It is the cleanest counter-example to “robots.txt covers all AI access.”

The per-bot product-token strings, the IP-range JSON endpoints, and the exact disallow recipes belong to the spoke entries linked in the table. The kept-fresh, audit-grade version — what to grep in logs, how to verify what reached you — belongs to AI Crawler Access Audit.

5. Declared control vs enforcement

RFC 9309 does not present itself as access control. The standard’s own preamble names the gap that the rest of this section unpacks: declared intent is not enforcement.

RFC 9309’s own disclaimer. §1 of the standard states: “the rules in a robots.txt file are not a form of access authorization.” Later: “This document is not a substitute for valid content security measures, and information that is not meant to be accessed should be properly secured.” Both lines are in the published standard (RFC 9309) — editorial commentary did not add the disclaimer; the standards body did.

User-agent is a self-asserted claim, not an identity. “I see GPTBot in my logs” is not “OpenAI fetched this.” A spoofed UA bypasses any rule keyed on it; a non-compliant crawler ignores the file entirely. Real identity is established by published IP ranges plus forward-confirmed reverse DNS — every major AI-crawler operator publishes IP lists for exactly this purpose. The verification mechanic itself belongs to the AI Crawler Access Audit playbook.

The observed compliance record. Documented behavior and observed behavior have an honest gap:

The bot-specific account of the Perplexity incidents routes to PerplexityBot; the principle stated here is what generalizes — and the broader access-decision account against which this protocol-mechanic section is the counterpart lives in AI Crawlers §5.

The position, plainly. Write the policy in robots.txt because compliant crawlers obey it — that is a real, useful effect for the documented first-party AI bots. But verifying that the policy is holding is the network layer’s job. Do not conflate the two layers; doing so is how a site quietly bleeds traffic to a crawler that ignored the file while believing it is protected.

6. Anti-patterns

Most robots.txt mistakes for AI bots fall into four pattern classes: precedence and group-merge errors, mechanism confusion (treating the file as enforcement, or as a meta-tag, or as rate limiting), location errors, and freshness errors. Each row below names a pattern that looks right and fails.

The load-bearing line. Most “robots.txt for AI” mistakes are category or precedence errors, not protocol-syntax errors. The syntax is forgiving; the consequences are not.

7. Three root files, three jobs

The single most common category error in AI publishing infrastructure is conflating three root-level files — robots.txt, sitemap.xml, llms.txt. They have non-overlapping jobs.

Three files, three jobs, none substitutes for another. robots.txt is no more “AI access control” than it was “Google access control” — it is the same file doing the same job, now addressed to a wider bot population with a categorical split. The change is in the audience, not in the file’s role.

8. Why this matters for GEO + how to act

Being reachable is necessary, not sufficient: robots.txt opens the gate to crawler access, and access in turn opens the gate to citation. Skip the file and most compliant AI bots assume nothing is restricted; write it wrong and the cost can be silent disappearance from AI answers.

Write the policy in robots.txt because compliant crawlers obey it; enforce it on the network layer because the rest do not. And remember: access is a gate, not a prize. Being reachable makes you a candidate — being citable is what wins the answer slot.

References

Standards & specifications:

• IETF — RFC 9309: Robots Exclusion Protocol
• Google Search Central — How Google Interprets the robots.txt Specification
• Google Search Central — A note on unsupported rules in robots.txt (the 2019 Noindex: retirement)

Vendor crawler documentation:

• OpenAI — Overview of OpenAI Crawlers · Publishers and Developers FAQ
• Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity — Perplexity Crawlers · How does Perplexity follow robots.txt?
• Google Search Central — Google’s common crawlers (including Google-Extended)
• Microsoft Bing — Which crawlers does Bing use?
• Apple — About Applebot
• Common Crawl — CCBot

Independent measurement & reporting:

• Cloudflare — From Googlebot to GPTBot: who’s crawling your site in 2025
• Cloudflare — Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
• TechCrunch — News outlets are accusing Perplexity of plagiarism and unethical web scraping
• Search Engine Land — Anthropic clarifies how Claude bots crawl sites and how to block them