robots.txt
Quick facts
- What it is
- Under RFC 9309, a plain-text file at the host root asks crawlers not to fetch specified paths. It is a voluntary request, not access authorization.
- Policy versus enforcement
- robots.txt states access intent. Network-layer controls are what enforce that intent against noncompliant or spoofed crawlers.
- Rules for AI bots
- Create named user-agent groups for each category instead of relying on one blanket
*block. A named bot reads only its own group, so*rules and per-botAllowrules do not merge. - Limits
- It does not grant access authorization, stop spoofed or noncompliant crawlers, or retract content already absorbed into model weights.
- Role in GEO
- It states crawler-access intent before network enforcement and long before citability matters. It is necessary, but not sufficient.
1. What robots.txt is
robots.txt is a plain-text file at the host root (/robots.txt) that lists the paths crawlers are asked not to fetch. Martijn Koster proposed the underlying Robots Exclusion Protocol in 1994. It remained a de facto convention for about 25 years before the IETF standardized it as RFC 9309 in September 2022.
Definition (GEO Wiki working definition): robots.txt is a UTF-8 plain-text file at
/robots.txtthat addresses crawlers by user-agent and asks them to skip certain paths. The request is nonbinding. It is the standardized expression of crawler-access intent, not access control.
RFC 9309 makes this limitation explicit: the rules “are not a form of access authorization,” and the protocol “is not a substitute for valid content security measures” (RFC 9309 §1). A robots.txt file declares intent; it does not enforce access.
The file itself has not changed for GEO, but the bot population has. Before AI crawlers, many publishers made one practical decision: whether to block Googlebot. Publishers now use the same file to address roughly 30 declared agents in three categories: training, retrieval, and user-triggered access. These categories can have different and sometimes opposite access consequences. AI Crawlers explains how they differ.
2. How the protocol works
A robots.txt file contains groups, each of which binds a set of rules to one or more user-agents. The file is served at the host root and uses a small, fixed grammar for matching paths.
The file. It must be a UTF-8 plain-text file, be scoped to one host, and be served from the literal path /robots.txt with a 2xx response. A 4xx response means that no rules have been declared, so the crawler may fetch any path (RFC 9309 §2.3.1.3). Operators handle a 5xx response conservatively: depending on their policy, their crawlers retry or temporarily treat the site as fully disallowed.
Records. A group contains one or more User-agent: lines followed by one or more Allow: and Disallow: rules. Blank lines separate groups, and comments begin with #.
Group selection. A crawler selects the single most specific group that matches its product token (RFC 9309 §2.2.1). Rules from other groups do not merge into the selected group. If the file contains a named group for a bot, the * group does not apply to that bot.
Path matching. The longest matching rule wins. When Allow and Disallow rules of equal path length conflict, Google documents that the least restrictive rule wins. In other words, Allow takes precedence over Disallow for equal-length matches (How Google Interprets the robots.txt Specification). RFC 9309 leaves this tie-breaking decision to the operator, and most major crawlers follow Google’s convention.
Wildcards. * matches zero or more characters, while $ anchors a match to the end of the URL. Google, Bing, OpenAI, Anthropic, and Perplexity support both symbols and evaluate them within each rule rather than as one combined regular expression.
Sitemap directive. Sitemap: https://... applies across the file rather than to a particular group. Sitemap & IndexNow explains how sitemap discovery works.
Here is how the precedence works:
User-agent: GPTBot
Disallow: /
Allow: /blog/
User-agent: *
Disallow: /private/GPTBot selects the GPTBot group, reads Disallow: / together with Allow: /blog/, and fetches nothing except /blog/. Every other bot selects the * group and is restricted only from /private/. The * rule does not apply to GPTBot because GPTBot has its own group.
3. How AI-bot categories map to robots.txt
Training, retrieval, and user-triggered agents can be represented as separate named user-agent groups. AI Crawlers explains why the categories require different access decisions; the examples below translate those decisions into robots.txt rules.
Opt out of training only. This preserves eligibility for live citations while excluding future training use.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /Retrieval and user-triggered fetchers stay untouched, so live ChatGPT, Claude, Perplexity, and Google AI Overviews answers can still cite you.
Opt out of retrieval. This means your site will no longer appear in current AI answers.
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Bingbot
Disallow: /The protocol cannot separate Google Search from AI Overviews. Blocking Googlebot removes a site from both because Google does not publish a separate “Search only, no AI Overviews” token. Google-Extended governs the training use of fetched content rather than crawling, so it appears in the training-only block above.
Allow everything by naming each bot explicitly:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /These rules have the same effect as an empty robots.txt file. Writing them explicitly can still document the intended policy for internal reviews and audit tools. It also preserves the named allowances if a downstream configuration layer prepends a blanket Disallow: / for *.
Named groups do not merge with *. A common mistake is to combine a blanket Disallow: / under * with per-bot Allow: lines in named groups. Each named bot reads only its own group, so the file does not create the merged policy that the writer intended.
Before implementing these examples, confirm the current product tokens and verification details for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, OAI-SearchBot, and ChatGPT-User. The AI Crawler Access Audit playbook explains how to search logs and verify which agents actually reached the site.
4. What each major operator documents about robots.txt compliance
Major operators describe different approaches to robots.txt. The table summarizes how their AI crawlers respond, including stated exceptions for user-initiated fetches.
| Operator | Tokens addressed | Documented robots.txt posture | Caveat |
|---|---|---|---|
| OpenAI | GPTBot · OAI-SearchBot · ChatGPT-User | GPTBot and OAI-SearchBot honor robots.txt as documented. As of late 2025, OpenAI’s bot documentation states that “because these actions are initiated by a user, robots.txt rules may not apply” to ChatGPT-User. | OpenAI made the user-triggered exception explicit in its December 2025 documentation revision (OpenAI bots). |
| Anthropic | ClaudeBot · Claude-SearchBot · Claude-User | Anthropic documents all three bots as honoring robots.txt. Its 2025 documentation introduced separate controls and disallow examples for each bot. | Anthropic also states that its bots honor the nonstandard Crawl-delay: extension alongside standard Disallow: rules (Anthropic crawler docs). |
| Perplexity | PerplexityBot · Perplexity-User | PerplexityBot respects robots.txt, so disallowing it prevents page text from being indexed. Perplexity documents Perplexity-User as user-initiated and says that robots.txt restrictions “generally do not apply.” | Perplexity has described this user-triggered exception in its own terms since 2024 (Perplexity Crawlers; How Perplexity follows robots.txt). |
| Googlebot · Google-Extended | Googlebot honors robots.txt. Google-Extended is a training-use opt-out token that governs whether fetched content is used to train Google’s generative AI models, not whether Googlebot crawls. | Blocking Googlebot removes a site from Search and AI Overviews together because Google provides no separate AIO opt-out (Google’s common crawlers). | |
| Apple | Applebot · Applebot-Extended | Both honor robots.txt. Applebot-Extended is a training-use opt-out for Apple Intelligence and other generative AI, while Applebot continues to serve Spotlight, Siri, and Safari. | Disallowing Applebot-Extended while allowing Applebot preserves visibility in Spotlight and Siri (About Applebot). |
| Microsoft (Bing) | Bingbot | Bingbot honors robots.txt and supports both Bing Search and Bing Copilot. | Unlike Google and Apple, Microsoft had not published a separate AI-training opt-out token as of 2026-05 (Which crawlers does Bing use?). |
| Common Crawl | CCBot | CCBot honors robots.txt and the nonstandard Crawl-delay: extension. | Common Crawl data indirectly supplies many third-party AI training corpora, so opting out can affect anyone who trains on Common Crawl dumps (CCBot). |
User-triggered fetches are a recurring exception. Operators describe ChatGPT-User, Claude-User, and Perplexity-User as user-initiated and generally outside standard robots.txt restrictions. Their stated reasoning is the same: a fetch prompted by one person asking about one URL acts as a proxy for human browsing rather than autonomous crawling. OpenAI stated this most explicitly in its late-2025 documentation revision, while Perplexity’s help page has used the same framing since 2024. These agents show why robots.txt does not cover every form of AI access.
Product-token strings, published IP-range endpoints, and disallow rules can change independently for each bot. The AI Crawler Access Audit provides the current procedure for finding these agents in logs and verifying which requests reached a site.
5. Declared control versus enforcement
RFC 9309 does not define robots.txt as access control. Its preamble makes the distinction clear: declaring intent is not the same as enforcing it.
RFC 9309 includes its own disclaimer. Section 1 states that “the rules in a robots.txt file are not a form of access authorization.” It later adds: “This document is not a substitute for valid content security measures, and information that is not meant to be accessed should be properly secured.” Both statements appear in the published standard (RFC 9309).
A user-agent is a self-reported claim, not proof of identity. Seeing GPTBot in a log does not prove that OpenAI fetched the page. A spoofed user-agent bypasses any rule that depends on the name, and a noncompliant crawler can ignore the file entirely. Published IP ranges and forward-confirmed reverse DNS establish identity; every major AI-crawler operator publishes IP lists for this purpose. The AI Crawler Access Audit playbook explains the verification process.
Published policies and observed behavior do not always match. The evidence supports only these conclusions:
| What the evidence shows | What can be concluded |
|---|---|
| Major first-party AI bots state that they honor robots.txt as documented. | Each operator is stating a policy, not providing a technical guarantee. Operators can also revise their policies, as the ChatGPT-User documentation change shows. |
| The protocol clearly expresses a site’s intent to compliant agents. | A well-written file cannot enforce that intent against a noncompliant or spoofed agent. |
| Independent measurement can show how many sites express an AI-crawler policy. | Cloudflare’s 2025 survey found that only about 14% of sampled domains had robots.txt files that targeted AI bots (Cloudflare, 2025-07-01). Most sites in the sample had expressed no AI policy. |
| First-party operators publish compliance policies. | Independent reporting has also documented noncompliance at scale. In 2024, several news publishers reported that a major answer engine fetched content from sites that had disallowed its crawler (TechCrunch, 2024-07-02). In 2025, Cloudflare reported that an undeclared crawler impersonated a normal browser and rotated IP addresses to evade no-crawl directives across tens of thousands of domains (Cloudflare, 2025-08-04). |
| A site can enforce crawler access. | Enforcement requires network-layer controls such as WAF rules, verified-bot allowlists, or request-signing schemes. robots.txt does not provide those controls. |
The incidents involving PerplexityBot show how noncompliance affected Perplexity. AI Crawlers compares the same enforcement problem across crawler categories and access decisions.
Use both policy and enforcement. Write the policy in robots.txt because documented first-party AI bots comply with it. Then verify at the network layer that the policy is working. A site that treats the file as enforcement may keep serving a crawler that ignores the rules while mistakenly assuming access is blocked.
6. Anti-patterns
Most robots.txt mistakes involving AI bots fall into four categories: precedence and group-merging errors, confusion about what the file can do, incorrect file placement, and outdated bot lists. The following patterns look reasonable but fail in practice.
| Anti-pattern | Why it looks right | Why it actually fails |
|---|---|---|
User-agent: * Disallow: / followed by per-bot Allow: in named groups | It appears to set a default and then admit trusted bots. | Each named bot selects only its own group (RFC 9309 §2.2.1). The * group does not apply to a bot with a named group. An Allow: rule in the GPTBot group therefore does not undo Disallow: / in the * group because the two groups never merge. |
Blocking GPTBot to stay out of ChatGPT | GPTBot is operated by OpenAI. | GPTBot is used only for training. Live ChatGPT search answers use OAI-SearchBot for indexing and ChatGPT-User for user-triggered fetches. As AI Crawlers explains, blocking the wrong category produces the opposite result from the one intended. |
| Treating robots.txt as enforcement against bad actors | A disallow rule appears to prevent access. | The protocol is voluntary, and RFC 9309 §1 says that its rules “are not a form of access authorization.” Spoofed or noncompliant agents can ignore the file. Enforcement requires a WAF or verified-bot allowlist (AI Crawler Access Audit). |
Crawl-delay: to rate-limit AI bots | The name makes it sound like a standard rate-limiting directive. | Crawl-delay: is not part of RFC 9309. Support is inconsistent: Common Crawl’s CCBot and Anthropic’s Claude bots document that they honor it, while Google states that Googlebot does not. Network-layer rate limits provide a dependable control across AI crawlers. |
Noindex: directives in robots.txt | robots.txt appears to control whether pages are indexed. | Noindex has never been a standard robots.txt directive. Google ended support for the unofficial Noindex: line on 2019-09-01 (A note on unsupported rules in robots.txt). Use <meta name="robots" content="noindex"> or the X-Robots-Tag HTTP header instead. |
robots.txt at /foo/robots.txt, or no separate file for a subdomain | The file exists at a location that may seem convenient. | RFC 9309 §2.3 requires a file at the literal /robots.txt path for each host. Subdomains and different schemes (http versus https) each need their own file. Crawlers ignore a robots.txt file at a non-root path. |
| A static AI-bot allowlist, never revisited | The list appears to admit only trusted crawlers. | The AI-crawler population grew from about 3 named bots in 2023 to about 30 in 2026. A static allowlist excludes every newly released bot by default, including retrieval bots whose exclusion immediately costs the site citations. |
| Disallowing the training crawler after the page already shipped | The new rule appears to remove the content from the training set. | robots.txt applies only to future crawling. It cannot retract content already absorbed into a prior training corpus. Any retraction mechanism, including an opt-out request form or takedown channel, depends on the vendor rather than the robots.txt file. |
Most robots.txt mistakes involving AI are category or precedence errors, not syntax errors. The syntax is forgiving, but selecting the wrong bot or misunderstanding how groups interact can have significant consequences.
7. Three root files, three jobs
Publishers often confuse three root-level files: robots.txt, sitemap.xml, and llms.txt. Their purposes do not overlap.
| File | What it does | What it does not do |
|---|---|---|
robots.txt | Crawler-access policy: asks whether a bot may fetch a path. It is standardized as RFC 9309. | It does not curate, render, rank, or enforce access. It is a request, not authorization. |
sitemap.xml | Discovery and completeness: lists the pages intended for indexing (Sitemap & IndexNow). | It does not curate, grant access, or signal quality. It is not a “best of” list. |
llms.txt | Curation and clean rendering: identifies selected pages and presents them in clean Markdown (llms.txt). | It does not grant or deny access, serve as a ranking signal, or provide a complete discovery file. |
None of these files substitutes for another. robots.txt is no more “AI access control” than it was “Google access control.” It performs the same function for a larger bot population divided into more categories. The audience changed; the file’s role did not.
8. How robots.txt affects GEO
A page must be reachable before an AI engine can cite it, but reachability alone does not make it citable. If a site publishes no robots.txt policy, most compliant AI bots assume that no paths are restricted. Incorrect rules can instead remove the site from AI answers without making the cause obvious.
| Goal | Recommended action |
|---|---|
| Decide the per-category access policy (training versus retrieval versus user-triggered) | Use AI Crawlers to assess the different consequences of blocking each category. |
| Write the actual robots.txt directives | Create separate named user-agent groups and apply the appropriate Allow: or Disallow: rules. |
| Get the per-bot UA strings, IP ranges, and verification recipes | GPTBot · ClaudeBot · PerplexityBot · Google-Extended · Applebot-Extended · OAI-SearchBot · ChatGPT-User |
| Verify what actually reached your site | Follow the AI Crawler Access Audit to compare declared policy with observed traffic. |
| Curate which pages an LLM should read first | Use llms.txt to identify selected pages and provide clean Markdown versions. |
| Make sure the page is easy to quote once fetched | Improve its citability so an engine can extract and attribute useful passages. |
| Put crawler access in a broader GEO strategy | Use Generative Engine Optimization to connect crawler access with other visibility decisions. |
Write the policy in robots.txt because compliant crawlers follow it, and enforce the policy at the network layer because other agents may not. Reachability makes a page eligible for consideration. Citability determines whether an engine can use it effectively in an answer.
References
Standards & specifications:
- IETF: RFC 9309: Robots Exclusion Protocol
- Google Search Central: How Google Interprets the robots.txt Specification
- Google Search Central: A note on unsupported rules in robots.txt (the 2019
Noindex:retirement)
Vendor crawler documentation:
- OpenAI: Overview of OpenAI Crawlers · Publishers and Developers FAQ
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity Crawlers · How does Perplexity follow robots.txt?
- Google Search Central: Google’s common crawlers (including Google-Extended)
- Microsoft Bing: Which crawlers does Bing use?
- Apple: About Applebot
- Common Crawl: CCBot
Independent measurement & reporting:
- Cloudflare: From Googlebot to GPTBot: who’s crawling your site in 2025
- Cloudflare: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
- TechCrunch: News outlets are accusing Perplexity of plagiarism and unethical web scraping
- Search Engine Land: Anthropic clarifies how Claude bots crawl sites and how to block them
Frequently asked questions
Does robots.txt actually stop AI crawlers?
Will blocking GPTBot remove me from ChatGPT search?
Why doesn't User-agent: * plus a per-bot Allow: grant access?
User-agent: * and Disallow: / in one group, then User-agent: GPTBot and Allow: /blog/ in another, GPTBot reads only the GPTBot group. It gets /blog/ and nothing else because the * group does not apply. The file can therefore look correct while producing the wrong policy.Is Crawl-delay: a valid robots.txt directive?
Crawl-delay: is not part of RFC 9309. Support varies by vendor: Common Crawl's CCBot and Anthropic's Claude bots document that they honor it, while Google states that Googlebot ignores it. Because support is inconsistent across AI crawlers, use network-layer rate limiting when you need a dependable control.Can robots.txt remove my content from a model that already trained on it?
See also
Sources
Primary
- RFC 9309: Robots Exclusion Protocol · IETF · 2022-09-01
- How Google Interprets the robots.txt Specification · Google Search Central
- A note on unsupported rules in robots.txt · Google Search Central · 2019-07-02
- Overview of OpenAI Crawlers (GPTBot / OAI-SearchBot / ChatGPT-User) · OpenAI
- Publishers and Developers FAQ — OpenAI Help Center · OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic
- Perplexity Crawlers (PerplexityBot / Perplexity-User) · Perplexity AI
- How does Perplexity follow robots.txt? · Perplexity AI
- Google's common crawlers (Google-Extended) · Google Search Central
- Which crawlers does Bing use? · Microsoft Bing
- About Applebot · Apple
- Common Crawl — CCBot · Common Crawl
Secondary
- From Googlebot to GPTBot: who's crawling your site in 2025 · Cloudflare
- Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives · Cloudflare
- News outlets are accusing Perplexity of plagiarism and unethical web scraping · TechCrunch
- Anthropic clarifies how Claude bots crawl sites and how to block them · Search Engine Land