GPTBot

1. What GPTBot is

GPTBot is the crawler OpenAI sends to collect publicly reachable content that may be used to train its foundation models. OpenAI’s own description is narrow and worth taking literally: it “is used to crawl content that may be used in training our generative AI foundation models” (see Overview of OpenAI Crawlers).

It fetches on a schedule of OpenAI’s own choosing, for one declared downstream use. It answers no question and serves no live response.

One property generates every difficulty on this page. The output of a GPTBot fetch is not a citation, a link, or an impression — it is a contribution to model weights that will not be attributable to you, will not arrive for months, and will never appear in any dashboard. Deciding about this bot means deciding about a benefit no one can observe.

It is the training member of a set that OAI-SearchBot (search indexing) and ChatGPT-User (live user-triggered fetch) complete, all four tokens operated by OpenAI. The categorical logic — why blocking one has nothing to do with blocking another — is worked out in AI crawlers.

The launch date matters here in a way it does not for its siblings. GPTBot arrived in August 2023 as the first mainstream training crawler shipping with a published opt-out token, which made it the default target of nearly every “block AI” recipe written since — including a great many that block the wrong thing.

2. Identifying GPTBot in your logs

The user-agent string OpenAI publishes:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Match the bare GPTBot token, never the version number. OpenAI presents this as an example and notes the version may change — and it has: the widely recirculated GPTBot/1.2 string that still appears in blog posts and rule templates is out of date. A rule pinned to a version fails silently the day the number moves.

One documented detail is easy to miss and useful when a policy change needs confirming. OpenAI states that “when fetching robots.txt files, we may add a robots.txt marker to the user-agent string to help site owners distinguish those requests from requests for other resources.” The exact marker text is not published, so treat this as a signal to look for rather than a string to match — a request for /robots.txt from a confirmed GPTBot address is the cheapest available evidence that a new directive was actually read.

GPTBot publishes its address ranges, and their shape is a cleanly verifiable structural fact. Counts below were fetched from OpenAI’s endpoints and cross-checked for overlap on 28 July 2026:

Two things fall out. GPTBot runs on a small footprint that shares six prefixes with the search crawler and none with the user-triggered fetcher. And the training crawler’s range file is comparatively static — it had not been regenerated in roughly nine months at the date above, while the user-triggered fetcher’s was five days old. That is an observation about publication cadence, not a claim about crawl activity.

The overlap has a documented explanation. OpenAI states: “If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling.” The vendor language and the address data agree, and that is as far as the evidence goes — it does not follow that blocking GPTBot raises OAI-SearchBot’s request volume, or that allowing both reduces server load. Neither claim has published support.

Verification is IP-based, plus forward-confirmed reverse DNS per the principle described in AI crawlers. A GPTBot string in a log proves nothing on its own; it is the most-spoofed token in this category precisely because it is the most famous. The runnable procedure — what to grep, how to confirm what actually reached you — is the AI crawler access audit.

3. The robots.txt opt-out, and what it does not reach

GPTBot is documented to honor robots.txt. OpenAI’s framing sentence names it explicitly: “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” The consequence is stated just as plainly — “disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”

This is the clean case. The user-triggered carve-out that complicates ChatGPT-User does not apply to a scheduled crawler.

The minimal directive:

User-agent: GPTBot
Disallow: /

A path-scoped policy can exclude a directory while keeping a public subset eligible:

User-agent: GPTBot
Disallow: /members/
Allow: /members/public-report/

The longer, more specific Allow wins for the public report. Group merging, path specificity, case sensitivity, and host scope all follow the ordinary rules covered in robots.txt — and the boundary between an allowed and a disallowed path is worth testing rather than assuming.

Propagation is not instant, and the published figure is narrower than it is usually quoted. OpenAI states that “for search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust.” That sentence is scoped to search results. OpenAI publishes no equivalent interval for the training crawl, so a specific propagation guarantee for GPTBot is not something the documentation supports.

Three limits deserve to be stated as limits rather than softened into caveats:

1. Not retroactive. A disallow governs future crawling. It does not remove content from models already trained, and no OpenAI mechanism does (§6).
2. Not enforcement. RFC 9309 states that its rules “are not a form of access authorization” and that the protocol “is not a substitute for valid content security measures.” Material that must not be read needs authentication, not a text file.
3. Not a corpus-wide opt-out. Disallowing GPTBot governs OpenAI’s own crawler. It says nothing about content already collected into third-party web corpora that model builders also draw on, and nothing about crawlers operated by anyone else.

On llms.txt: it is not documented as a mechanism for expressing training permission, and publishing one does not change GPTBot’s behavior.

4. What blocking GPTBot does — and what it does not

Blocking GPTBot does not remove you from ChatGPT. Live answers are served by OAI-SearchBot, which builds the search index, and ChatGPT-User, which fetches a page while someone is asking about it. GPTBot participates in neither. A site can disallow it and remain fully citable in ChatGPT.

The mirror-image error costs more and is less discussed: disallowing OAI-SearchBot “to stop AI taking my content” surrenders citation immediately to prevent a training exposure that a different token governs.

The same training-versus-retrieval split governs the decision for ClaudeBot, Anthropic’s training crawler. Google-Extended reaches a similar outcome through a structurally different device — a control token with no crawler and no user agent of its own. PerplexityBot sits on the retrieval side entirely, which is why it is not the analogous decision at all.

5. Is there a GEO payoff to allowing it?

Every sibling crawler can point at a citation consequence. This one cannot, and being precise about that is more useful than manufacturing a reason.

Two mechanisms get conflated, and separating them settles most of the argument:

The mechanism in the second row is real. Content absorbed during training can contribute to what a model holds about an entity, which is the same currency brand mentions trade in — being known without being linked.

But a mechanism is not a measured effect, and here the gap is structural rather than merely unstudied. Language models are, in the words of the research on knowledge attribution, “often oblivious to the source(s) of such knowledge” — attributing a model’s parametric output back to the documents that produced it requires purpose-built training-time intervention, not analysis after the fact (Khalifa et al., 2024). The attribution you would need in order to prove a GPTBot payoff is precisely the thing standard-trained models do not support.

So the honest statement is: no published study isolates “this site allowed GPTBot” as a variable and demonstrates a citation or mention lift. The counterfactual is unavailable by construction — you cannot A/B a training run, cannot observe your content in weights, and cannot attribute an unsourced mention to any particular crawl.

Which way the resulting asymmetry points is not toward reflexive blocking. The cost of allowing is diffuse: bandwidth, and a licensing position not taken. The cost of the panic-block that usually accompanies the decision is specific and immediate, because the same edit routinely catches the retrieval crawler too. The bot with the unmeasurable payoff is not the bot that most block decisions actually harm.

One more piece of scope honesty. Many publishers block GPTBot for reasons that are not GEO reasons at all — licensing leverage, rights policy, legal posture, infrastructure cost. Those are legitimate grounds that a visibility analysis is not equipped to score, and they are frequently the right grounds. What can be said is narrower: the decision should not be made on an expected citation benefit, in either direction, because none has been demonstrated. The visibility question that does have answers belongs to the retrieval side, which is what generative engine optimization is about.

6. The compensation layer that grew around this bot

GPTBot is the specific token the “should publishers be paid” conversation attached itself to, and anyone deciding today is deciding inside that context.

Media Manager did not ship. On 7 May 2024 OpenAI announced a tool to let creators identify their works and specify inclusion or exclusion from AI training, stating: “The goal is to have the tool in place by 2025” (TechCrunch). The deadline passed without delivery (TechCrunch, 1 January 2025), and as of this entry’s verification date OpenAI has announced no launch. The consequence is the load-bearing part: robots.txt remains the only opt-out mechanism that actually exists.

Alongside it runs a negotiated path. OpenAI has signed content licensing agreements with a substantial number of publishers, which is a real alternative to the token — but not one available at most scales. For everyone outside those deals, the directive in §3 is the entire toolkit.

The layer that has moved fastest is the network. Cloudflare now ships defaults that do what robots.txt only requests, and a dated change is worth planning around: from 15 September 2026, “Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default” (Cloudflare). GPTBot is a single-purpose training crawler, so it falls squarely in the blocked category — no mixed-purpose ambiguity applies. The defaults land on newly onboarded domains; existing customers can set their preference beforehand. Cloudflare also offers a managed robots.txt that writes AI-training directives on a site’s behalf (Cloudflare, 1 July 2025).

The direction, stated no further than the evidence allows: control over this bot is migrating from a voluntary text file toward the network layer and toward contracts.

7. How much of the web blocks it, and how much it crawls

Every figure here depends on its denominator, and GPTBot statistics are the most-quoted and most-garbled numbers in this field. A curated list of major news sites and a broad sample of ordinary domains produce answers that differ several-fold about the same bot, and both are correct about their own population.

The spread is the finding:

Those are not in conflict; they answer different questions. Cloudflare supplies the fact that reconciles them: only about 37% of the top 10,000 domains have a robots.txt file at all. Most of the web expresses no policy on this bot in either direction, so any headline of the form “X% of the web blocks GPTBot” is meaningless without naming what was counted.

What is robust across sources is the direction: blocking rose substantially from GPTBot’s August 2023 launch through 2025, concentrated among large publishers rather than distributed evenly.

On volume, one careful reading beats several loose ones. Across 2025 on Cloudflare’s network, GPTBot accounted for about 7.5% of verified bot traffic, against Googlebot’s more than 28% (Cloudflare Radar 2025 Year in Review). Training was the dominant purpose throughout — near 80% of AI bot crawling in a July 2025 measurement window (Cloudflare) — while user-action crawling grew more than twentyfold over the year from a very small base.

Two disciplines follow for anyone citing numbers on this bot. Write 2025 figures in the past tense: per-bot monthly rankings moved repeatedly through 2026 and competing operators’ crawlers have led at different points, so “GPTBot is the largest AI crawler” is not a claim current data supports — and on the network measured above it was never the largest at all. And treat month-to-month per-bot rankings from third-party trackers with care; they disagree with each other, use undisclosed populations, and are frequently quoted long after the month they describe.

8. Choosing a policy

Record the outcome as policy rather than as a file edit: intended uses, responsible owner, affected hosts and schemes, the exact directives, deployment date, and a review date. Implementation evidence — what the origin actually serves, and what actually reached it — comes from the AI crawler access audit.

Set the review date deliberately. The token itself has been stable since 2023; everything around it has not. The opt-out tool promised for 2025 never arrived, a CDN default flips in September 2026, and the licensing layer changed shape twice in two years. A decision made once and never revisited is a decision made against a landscape that no longer exists.

9. Related entries

• AI crawlers — the three-category model and the per-category access decision
• OAI-SearchBot — OpenAI’s search indexer, and the correct lever for Search inclusion
• ChatGPT-User — OpenAI’s user-triggered fetcher, and why robots.txt may not bind it
• ClaudeBot — Anthropic’s training crawler, the same decision at another operator
• Google-Extended — a training control implemented as a token with no crawler
• PerplexityBot — the retrieval-side comparison
• robots.txt — the protocol, and how the directives are parsed
• llms.txt — what it is, and what it does not control
• OpenAI — the operator behind all four tokens
• Brand mentions — the entity-prior currency parametric recall would trade in
• Generative engine optimization — the visibility question the retrieval side answers
• AI crawler access audit — verifying what actually reached you
• ChatGPT Search — the engine surface GPTBot does not feed

References

Primary

• OpenAI — Overview of OpenAI Crawlers
• OpenAI — GPTBot published IP ranges · OAI-SearchBot ranges · ChatGPT-User ranges
• IETF — RFC 9309: Robots Exclusion Protocol
• Cloudflare — Your site, your rules: new AI traffic options for all customers (1 July 2026)
• Cloudflare — Control content use for AI training with managed robots.txt (1 July 2025)
• Khalifa, Wadden, Strubell, Lee, Wang, Beltagy, Peng — Source-Aware Training Enables Knowledge Attribution in Language Models (April 2024)
• Reuters Institute — How many news websites block AI crawlers?

Secondary

• Cloudflare — The 2025 Cloudflare Radar Year in Review
• Cloudflare — A deeper look at AI crawlers: traffic by purpose and industry (28 August 2025)
• TechCrunch — OpenAI says it’s building a tool to let content creators opt out of AI training (7 May 2024)
• TechCrunch — OpenAI failed to deliver the opt-out tool it promised by 2025 (1 January 2025)