GPTBot

1. What GPTBot is

GPTBot is the crawler OpenAI sends to collect publicly reachable content that may be used to train its foundation models. OpenAI’s own description is narrow and worth taking literally: it “is used to crawl content that may be used in training our generative AI foundation models” (see Overview of OpenAI Crawlers).

GPTBot crawls on OpenAI’s schedule for one stated purpose: training. It does not answer questions or serve live responses.

A GPTBot visit does not produce a citation, link, or impression. Instead, the content may contribute to model weights months later. That contribution cannot be attributed to the site and will not appear in a dashboard, so any benefit from allowing the crawler remains unobservable.

GPTBot is OpenAI’s training crawler within its four-token system. OAI-SearchBot handles search indexing, and ChatGPT-User handles live, user-triggered fetches. OpenAI operates all four tokens. AI crawlers explains why blocking one category does not block the others.

OpenAI launched GPTBot in August 2023 as the first mainstream training crawler with a published opt-out token. It consequently became the default target of many “block AI” instructions, including instructions that block the wrong crawler.

2. Identifying GPTBot in your logs

OpenAI publishes this user-agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Match the GPTBot token without the version number. OpenAI presents the string as an example and notes that the version may change. It already has: the widely circulated GPTBot/1.2 string still found in blog posts and rule templates is outdated. A version-specific rule will stop matching when the number changes.

OpenAI also states that “when fetching robots.txt files, we may add a robots.txt marker to the user-agent string to help site owners distinguish those requests from requests for other resources.” It does not publish the exact marker text, so use the marker as a signal rather than a value to match. A request for /robots.txt from a confirmed GPTBot address is the simplest available evidence that the crawler read a new directive.

OpenAI publishes address ranges for GPTBot. The counts below were fetched from OpenAI’s endpoints and checked for overlap on 28 July 2026:

GPTBot uses a relatively small published address set. It shares six prefixes with the search crawler and none with the user-triggered fetcher. Its range file had not been regenerated for roughly nine months as of the comparison date, while the ChatGPT-User file was five days old. These dates show when OpenAI generated the files, not how often the crawlers were active.

OpenAI documents why the two bots may share infrastructure: “If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling.” The published address data is consistent with that statement. It does not establish that blocking GPTBot increases OAI-SearchBot traffic or that allowing both bots reduces server load. Neither claim has published support.

Verify GPTBot by IP address and forward-confirmed reverse DNS, following the principle described in AI crawlers. A GPTBot string in a log proves nothing by itself. As the best-known token in this category, it is also the most frequently spoofed. Use the AI crawler access audit to find and verify requests.

3. The robots.txt opt-out and its limits

OpenAI documents GPTBot as honoring robots.txt: “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” It also states that “disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”

Because GPTBot is a scheduled crawler, the user-triggered exception that applies to ChatGPT-User does not apply here.

The minimal directive:

User-agent: GPTBot
Disallow: /

A path-specific policy can exclude a directory while allowing a public subset:

User-agent: GPTBot
Disallow: /members/
Allow: /members/public-report/

The longer, more specific Allow rule takes precedence for the public report. Group merging, path specificity, case sensitivity, and host scope follow the standard rules described in robots.txt. Test both an allowed path and a disallowed path to confirm the boundary.

Changes may not take effect immediately. OpenAI states that “for search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust.” That estimate applies specifically to search results. OpenAI publishes no equivalent interval for GPTBot, so its documentation does not support a specific propagation time for the training crawler.

The opt-out has three important limits:

1. Not retroactive. A disallow governs future crawling. It does not remove content from models already trained, and no OpenAI mechanism does (§6).
2. Not enforcement. RFC 9309 states that its rules “are not a form of access authorization” and that the protocol “is not a substitute for valid content security measures.” Content that must remain private requires authentication rather than a text file.
3. Not a corpus-wide opt-out. Disallowing GPTBot applies only to OpenAI’s crawler. It does not affect content already collected in third-party web corpora used by model developers, nor does it affect crawlers operated by other organizations.

llms.txt is not documented as a way to grant or deny training permission. Publishing the file does not change GPTBot’s behavior.

4. What happens when you block GPTBot

Blocking GPTBot does not remove a site from ChatGPT. OAI-SearchBot builds the search index for live answers, while ChatGPT-User fetches pages in response to user requests. GPTBot does neither. A site can disallow GPTBot and remain eligible for citation in ChatGPT.

Blocking OAI-SearchBot instead has an immediate cost. Disallowing OAI-SearchBot “to stop AI taking my content” gives up citations while leaving the training decision to a different token.

The same distinction between training and retrieval applies to ClaudeBot, Anthropic’s training crawler. Google-Extended provides a comparable training control through a different mechanism: it is a control token without a crawler or user agent of its own. PerplexityBot is a retrieval crawler, so blocking it has different consequences.

5. Is there a GEO payoff to allowing it?

Retrieval crawlers have a direct effect on citation eligibility. No comparable citation effect has been demonstrated for GPTBot.

The distinction depends on two different mechanisms:

The second mechanism is real, but its effect on any one site cannot be measured. Content absorbed during training can contribute to a model’s prior knowledge about an entity. Brand mentions may work similarly by helping an entity become known without requiring a link.

A plausible mechanism is not the same as a measured effect. Research on knowledge attribution describes language models as “often oblivious to the source(s) of such knowledge” and explains that tracing parametric output back to the documents that produced it requires a purpose-built intervention during training, not analysis after training (Khalifa et al., 2024). Standard-trained models therefore do not provide the source attribution needed to prove a benefit from allowing GPTBot.

No published study has isolated “this site allowed GPTBot” as a variable and demonstrated an increase in citations or mentions. The necessary comparison cannot be observed: a site cannot A/B a training run, inspect its content in model weights, or attribute an unsourced mention to a particular crawl.

The costs on each side are different. Allowing GPTBot uses bandwidth and forgoes the opportunity to use crawler access as part of a licensing position. Broad blocking rules often catch the retrieval crawler as well, creating an immediate and measurable cost. GPTBot’s possible benefit remains unmeasurable, while the loss of citations from blocking retrieval is direct.

Publishers may still block GPTBot for licensing leverage, rights policies, legal considerations, or infrastructure costs. Those are legitimate reasons, but a visibility analysis cannot assign their value. The decision should not be based on an expected citation effect in either direction because none has been demonstrated. Generative engine optimization addresses the visibility effects of retrieval instead.

6. Publisher compensation and access controls

GPTBot became the main technical focus of the debate over whether publishers should be paid for training content. That debate now shapes decisions about whether to allow the crawler.

Media Manager did not ship. On 7 May 2024, OpenAI announced a tool that would let creators identify their work and specify whether it could be used for AI training. The company stated, “The goal is to have the tool in place by 2025” (TechCrunch). That deadline passed without delivery (TechCrunch, 1 January 2025). As of 28 July 2026, OpenAI had not announced a launch. robots.txt therefore remains the only available opt-out mechanism.

OpenAI has also signed content licensing agreements with a substantial number of publishers. Those agreements provide a negotiated alternative to managing training access through the GPTBot token, but they are not an option for most publishers. For sites without such an agreement, the directive in §3 is the only available option.

Network-level controls have changed more quickly. Starting on 15 September 2026, Cloudflare states that “Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default” (Cloudflare). GPTBot is a single-purpose training crawler, so it falls within the blocked category. The defaults apply to newly onboarded domains, while existing customers can choose a preference before the change. Cloudflare also offers managed robots.txt controls that publish AI-training directives for a site (Cloudflare, 1 July 2025).

Together, these changes show that control over GPTBot is moving beyond a voluntary text file to network-level controls and contracts.

7. How widely GPTBot is blocked and how much traffic it generates

Every GPTBot statistic depends on what was measured. A curated list of major news sites and a broad sample of ordinary domains can produce results that differ severalfold, even when both accurately describe their respective populations.

The difference between two published measurements illustrates the problem:

The figures do not conflict because they answer different questions. Cloudflare also reported that only about 37% of the top 10,000 domains had a robots.txt file. Most sites expressed no policy on GPTBot in either direction. A claim that “X% of the web blocks GPTBot” is therefore incomplete unless it identifies the population and denominator.

Across sources, one trend remained consistent: blocking increased substantially between GPTBot’s August 2023 launch and 2025, with large publishers accounting for a disproportionate share.

Traffic volume requires the same care. Across Cloudflare’s network in 2025, GPTBot accounted for about 7.5% of verified bot traffic, compared with more than 28% for Googlebot (Cloudflare Radar 2025 Year in Review). Training accounted for nearly 80% of AI bot crawling during a July 2025 measurement window (Cloudflare). User-action crawling grew more than twentyfold over the year, though it started from a very small base.

Use the past tense when citing 2025 figures. Monthly rankings changed repeatedly during 2026, with crawlers from different operators taking the lead at different times. Current data therefore does not support describing GPTBot as the largest AI crawler, and it was never the largest crawler in Cloudflare’s 2025 network data. Third-party monthly rankings also require caution because trackers disagree, often do not disclose their populations, and may be quoted long after the month they measured.

8. Choosing a policy

Document the outcome as a policy, including intended uses, the responsible owner, affected hosts and schemes, exact directives, the deployment date, and a review date. Then use the AI crawler access audit to verify what the origin serves and which crawlers reach it.

Choose the review date deliberately. The token has remained stable since 2023, but the surrounding policies have changed. The opt-out tool promised for 2025 never arrived, a CDN default changes in September 2026, and licensing arrangements changed twice in two years. A policy that is never reviewed can quickly become outdated.

9. Related entries

• AI crawlers: the three crawler categories and the access decision for each one
• OAI-SearchBot: OpenAI’s search indexer and the control for ChatGPT Search inclusion
• ChatGPT-User: OpenAI’s user-triggered fetcher and the limits of robots.txt for those requests
• ClaudeBot: Anthropic’s training crawler and the corresponding decision for another operator
• Google-Extended: a training control implemented as a token without its own crawler
• PerplexityBot: a comparable crawler on the retrieval side
• robots.txt: the protocol and how its directives are parsed
• llms.txt: what the file does and does not control
• OpenAI: the operator behind all four tokens
• Brand mentions: how entity knowledge may form without links
• Generative engine optimization: how retrieval affects visibility in AI answers
• AI crawler access audit: how to verify which crawlers reached a site
• ChatGPT Search: the product surface that GPTBot does not support

References

Primary

• OpenAI: Overview of OpenAI Crawlers
• OpenAI: GPTBot published IP ranges · OAI-SearchBot ranges · ChatGPT-User ranges
• IETF: RFC 9309: Robots Exclusion Protocol
• Cloudflare: Your site, your rules: new AI traffic options for all customers (1 July 2026)
• Cloudflare: Control content use for AI training with managed robots.txt (1 July 2025)
• Khalifa, Wadden, Strubell, Lee, Wang, Beltagy, and Peng: Source-Aware Training Enables Knowledge Attribution in Language Models (April 2024)
• Reuters Institute: How many news websites block AI crawlers?

Secondary

• Cloudflare: The 2025 Cloudflare Radar Year in Review
• Cloudflare: A deeper look at AI crawlers: traffic by purpose and industry (28 August 2025)
• TechCrunch: OpenAI says it’s building a tool to let content creators opt out of AI training (7 May 2024)
• TechCrunch: OpenAI failed to deliver the opt-out tool it promised by 2025 (1 January 2025)