Skip to content

GPTBot

Quick facts

What it is
OpenAI's training crawler — it collects content that may be used to train future foundation models, and serves no live response
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot — match the bare token, never the version
robots.txt
Documented as manageable. OpenAI states that 'disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models'
Effect on ChatGPT Search
None. Search inclusion is governed by OAI-SearchBot; live lookups by ChatGPT-User. Blocking GPTBot costs no citation
The honest catch
No published evidence shows that allowing GPTBot lifts citations or mentions. The benefit is a mechanism, not a measured effect — and it is unmeasurable by construction

1. What GPTBot is

GPTBot is the crawler OpenAI sends to collect publicly reachable content that may be used to train its foundation models. OpenAI’s own description is narrow and worth taking literally: it “is used to crawl content that may be used in training our generative AI foundation models” (see Overview of OpenAI Crawlers).

It fetches on a schedule of OpenAI’s own choosing, for one declared downstream use. It answers no question and serves no live response.

One property generates every difficulty on this page. The output of a GPTBot fetch is not a citation, a link, or an impression — it is a contribution to model weights that will not be attributable to you, will not arrive for months, and will never appear in any dashboard. Deciding about this bot means deciding about a benefit no one can observe.

It is the training member of a set that OAI-SearchBot (search indexing) and ChatGPT-User (live user-triggered fetch) complete, all four tokens operated by OpenAI. The categorical logic — why blocking one has nothing to do with blocking another — is worked out in AI crawlers.

The launch date matters here in a way it does not for its siblings. GPTBot arrived in August 2023 as the first mainstream training crawler shipping with a published opt-out token, which made it the default target of nearly every “block AI” recipe written since — including a great many that block the wrong thing.

2. Identifying GPTBot in your logs

The user-agent string OpenAI publishes:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Match the bare GPTBot token, never the version number. OpenAI presents this as an example and notes the version may change — and it has: the widely recirculated GPTBot/1.2 string that still appears in blog posts and rule templates is out of date. A rule pinned to a version fails silently the day the number moves.

One documented detail is easy to miss and useful when a policy change needs confirming. OpenAI states that “when fetching robots.txt files, we may add a robots.txt marker to the user-agent string to help site owners distinguish those requests from requests for other resources.” The exact marker text is not published, so treat this as a signal to look for rather than a string to match — a request for /robots.txt from a confirmed GPTBot address is the cheapest available evidence that a new directive was actually read.

GPTBot publishes its address ranges, and their shape is a cleanly verifiable structural fact. Counts below were fetched from OpenAI’s endpoints and cross-checked for overlap on 28 July 2026:

TokenPublished IP filePrefixesFile generatedOverlap
GPTBotgptbot.json212025-10-306 shared with OAI-SearchBot
OAI-SearchBotsearchbot.json352026-01-026 shared with GPTBot
ChatGPT-Userchatgpt-user.json2862026-07-23none with either

Two things fall out. GPTBot runs on a small footprint that shares six prefixes with the search crawler and none with the user-triggered fetcher. And the training crawler’s range file is comparatively static — it had not been regenerated in roughly nine months at the date above, while the user-triggered fetcher’s was five days old. That is an observation about publication cadence, not a claim about crawl activity.

The overlap has a documented explanation. OpenAI states: “If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling.” The vendor language and the address data agree, and that is as far as the evidence goes — it does not follow that blocking GPTBot raises OAI-SearchBot’s request volume, or that allowing both reduces server load. Neither claim has published support.

Verification is IP-based, plus forward-confirmed reverse DNS per the principle described in AI crawlers. A GPTBot string in a log proves nothing on its own; it is the most-spoofed token in this category precisely because it is the most famous. The runnable procedure — what to grep, how to confirm what actually reached you — is the AI crawler access audit.

3. The robots.txt opt-out, and what it does not reach

GPTBot is documented to honor robots.txt. OpenAI’s framing sentence names it explicitly: “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” The consequence is stated just as plainly — “disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”

This is the clean case. The user-triggered carve-out that complicates ChatGPT-User does not apply to a scheduled crawler.

The minimal directive:

User-agent: GPTBot
Disallow: /

A path-scoped policy can exclude a directory while keeping a public subset eligible:

User-agent: GPTBot
Disallow: /members/
Allow: /members/public-report/

The longer, more specific Allow wins for the public report. Group merging, path specificity, case sensitivity, and host scope all follow the ordinary rules covered in robots.txt — and the boundary between an allowed and a disallowed path is worth testing rather than assuming.

Propagation is not instant, and the published figure is narrower than it is usually quoted. OpenAI states that “for search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust.” That sentence is scoped to search results. OpenAI publishes no equivalent interval for the training crawl, so a specific propagation guarantee for GPTBot is not something the documentation supports.

Three limits deserve to be stated as limits rather than softened into caveats:

  1. Not retroactive. A disallow governs future crawling. It does not remove content from models already trained, and no OpenAI mechanism does (§6).
  2. Not enforcement. RFC 9309 states that its rules “are not a form of access authorization” and that the protocol “is not a substitute for valid content security measures.” Material that must not be read needs authentication, not a text file.
  3. Not a corpus-wide opt-out. Disallowing GPTBot governs OpenAI’s own crawler. It says nothing about content already collected into third-party web corpora that model builders also draw on, and nothing about crawlers operated by anyone else.

On llms.txt: it is not documented as a mechanism for expressing training permission, and publishing one does not change GPTBot’s behavior.

4. What blocking GPTBot does — and what it does not

Blocking GPTBot does not remove you from ChatGPT. Live answers are served by OAI-SearchBot, which builds the search index, and ChatGPT-User, which fetches a page while someone is asking about it. GPTBot participates in neither. A site can disallow it and remain fully citable in ChatGPT.

The mirror-image error costs more and is less discussed: disallowing OAI-SearchBot “to stop AI taking my content” surrenders citation immediately to prevent a training exposure that a different token governs.

The moveThe belief behind itWhat actually happens
Block GPTBot to leave ChatGPT”It’s OpenAI’s crawler”No effect on ChatGPT Search or live answers — those are OAI-SearchBot and ChatGPT-User
Block GPTBot to undo past training”Opt out means opt out”Governs future crawls only; content in already-trained models is unaffected
Blanket Disallow: / for every AI bot”Protect the content”Surrenders citation to stop training — opposite categories with opposite costs
Allow GPTBot expecting more citations”Being in the model helps”No published evidence supports a citation lift (§5)
Treat the disallow as a licensing position”I’ve reserved my rights”A voluntary request, not a legal instrument or an enforcement layer

The same training-versus-retrieval split governs the decision for ClaudeBot, Anthropic’s training crawler. Google-Extended reaches a similar outcome through a structurally different device — a control token with no crawler and no user agent of its own. PerplexityBot sits on the retrieval side entirely, which is why it is not the analogous decision at all.

5. Is there a GEO payoff to allowing it?

Every sibling crawler can point at a citation consequence. This one cannot, and being precise about that is more useful than manufacturing a reason.

Two mechanisms get conflated, and separating them settles most of the argument:

MechanismWhat it producesWhich bot governs itMeasurable?
Retrieval groundingA cited answer, nowOAI-SearchBot / ChatGPT-UserYes — trackable per answer
Parametric recallThe model’s unsourced prior about youGPTBot, among other corporaNo — not attributable

The mechanism in the second row is real. Content absorbed during training can contribute to what a model holds about an entity, which is the same currency brand mentions trade in — being known without being linked.

But a mechanism is not a measured effect, and here the gap is structural rather than merely unstudied. Language models are, in the words of the research on knowledge attribution, “often oblivious to the source(s) of such knowledge” — attributing a model’s parametric output back to the documents that produced it requires purpose-built training-time intervention, not analysis after the fact (Khalifa et al., 2024). The attribution you would need in order to prove a GPTBot payoff is precisely the thing standard-trained models do not support.

So the honest statement is: no published study isolates “this site allowed GPTBot” as a variable and demonstrates a citation or mention lift. The counterfactual is unavailable by construction — you cannot A/B a training run, cannot observe your content in weights, and cannot attribute an unsourced mention to any particular crawl.

Which way the resulting asymmetry points is not toward reflexive blocking. The cost of allowing is diffuse: bandwidth, and a licensing position not taken. The cost of the panic-block that usually accompanies the decision is specific and immediate, because the same edit routinely catches the retrieval crawler too. The bot with the unmeasurable payoff is not the bot that most block decisions actually harm.

One more piece of scope honesty. Many publishers block GPTBot for reasons that are not GEO reasons at all — licensing leverage, rights policy, legal posture, infrastructure cost. Those are legitimate grounds that a visibility analysis is not equipped to score, and they are frequently the right grounds. What can be said is narrower: the decision should not be made on an expected citation benefit, in either direction, because none has been demonstrated. The visibility question that does have answers belongs to the retrieval side, which is what generative engine optimization is about.

6. The compensation layer that grew around this bot

GPTBot is the specific token the “should publishers be paid” conversation attached itself to, and anyone deciding today is deciding inside that context.

Media Manager did not ship. On 7 May 2024 OpenAI announced a tool to let creators identify their works and specify inclusion or exclusion from AI training, stating: “The goal is to have the tool in place by 2025” (TechCrunch). The deadline passed without delivery (TechCrunch, 1 January 2025), and as of this entry’s verification date OpenAI has announced no launch. The consequence is the load-bearing part: robots.txt remains the only opt-out mechanism that actually exists.

Alongside it runs a negotiated path. OpenAI has signed content licensing agreements with a substantial number of publishers, which is a real alternative to the token — but not one available at most scales. For everyone outside those deals, the directive in §3 is the entire toolkit.

The layer that has moved fastest is the network. Cloudflare now ships defaults that do what robots.txt only requests, and a dated change is worth planning around: from 15 September 2026, “Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default” (Cloudflare). GPTBot is a single-purpose training crawler, so it falls squarely in the blocked category — no mixed-purpose ambiguity applies. The defaults land on newly onboarded domains; existing customers can set their preference beforehand. Cloudflare also offers a managed robots.txt that writes AI-training directives on a site’s behalf (Cloudflare, 1 July 2025).

The direction, stated no further than the evidence allows: control over this bot is migrating from a voluntary text file toward the network layer and toward contracts.

7. How much of the web blocks it, and how much it crawls

Every figure here depends on its denominator, and GPTBot statistics are the most-quoted and most-garbled numbers in this field. A curated list of major news sites and a broad sample of ordinary domains produce answers that differ several-fold about the same bot, and both are correct about their own population.

The spread is the finding:

ReadingPopulationFigure
Reuters Institute, end of 202315 most-used news sites × 10 countries48% blocked OpenAI’s crawlers — ranging from 79% (US) to 20% (Mexico, Poland)
Cloudflare, 1 July 2025robots.txt files among top domains7.8% disallowed GPTBot

Those are not in conflict; they answer different questions. Cloudflare supplies the fact that reconciles them: only about 37% of the top 10,000 domains have a robots.txt file at all. Most of the web expresses no policy on this bot in either direction, so any headline of the form “X% of the web blocks GPTBot” is meaningless without naming what was counted.

What is robust across sources is the direction: blocking rose substantially from GPTBot’s August 2023 launch through 2025, concentrated among large publishers rather than distributed evenly.

On volume, one careful reading beats several loose ones. Across 2025 on Cloudflare’s network, GPTBot accounted for about 7.5% of verified bot traffic, against Googlebot’s more than 28% (Cloudflare Radar 2025 Year in Review). Training was the dominant purpose throughout — near 80% of AI bot crawling in a July 2025 measurement window (Cloudflare) — while user-action crawling grew more than twentyfold over the year from a very small base.

Two disciplines follow for anyone citing numbers on this bot. Write 2025 figures in the past tense: per-bot monthly rankings moved repeatedly through 2026 and competing operators’ crawlers have led at different points, so “GPTBot is the largest AI crawler” is not a claim current data supports — and on the network measured above it was never the largest at all. And treat month-to-month per-bot rankings from third-party trackers with care; they disagree with each other, use undisclosed populations, and are frequently quoted long after the month they describe.

8. Choosing a policy

Publisher intentDirectiveWhat it costs
Stay citable, decline trainingDisallow GPTBot; leave OAI-SearchBot and ChatGPT-User allowedNothing on the citation side; forfeits a parametric-recall benefit that is unmeasured
Permit bothNo GPTBot disallowContent stays eligible for the training use OpenAI names
Preserve a licensing positionDisallow GPTBot, plus network-layer controls and contractsrobots.txt alone is a request, not a reservation of rights
Path-scoped policyDisallow GPTBot on a subtree, allow a public subsetRequires testing the boundary, per robots.txt
Leave AI answers entirelyA different decision — the retrieval tokens, not this oneImmediate and total citation loss, per AI crawlers

Record the outcome as policy rather than as a file edit: intended uses, responsible owner, affected hosts and schemes, the exact directives, deployment date, and a review date. Implementation evidence — what the origin actually serves, and what actually reached it — comes from the AI crawler access audit.

Set the review date deliberately. The token itself has been stable since 2023; everything around it has not. The opt-out tool promised for 2025 never arrived, a CDN default flips in September 2026, and the licensing layer changed shape twice in two years. A decision made once and never revisited is a decision made against a landscape that no longer exists.

  • AI crawlers — the three-category model and the per-category access decision
  • OAI-SearchBot — OpenAI’s search indexer, and the correct lever for Search inclusion
  • ChatGPT-User — OpenAI’s user-triggered fetcher, and why robots.txt may not bind it
  • ClaudeBot — Anthropic’s training crawler, the same decision at another operator
  • Google-Extended — a training control implemented as a token with no crawler
  • PerplexityBot — the retrieval-side comparison
  • robots.txt — the protocol, and how the directives are parsed
  • llms.txt — what it is, and what it does not control
  • OpenAI — the operator behind all four tokens
  • Brand mentions — the entity-prior currency parametric recall would trade in
  • Generative engine optimization — the visibility question the retrieval side answers
  • AI crawler access audit — verifying what actually reached you
  • ChatGPT Search — the engine surface GPTBot does not feed

References

Primary

Secondary

Frequently asked questions

What is GPTBot?
GPTBot is the crawler OpenAI sends to collect publicly reachable content that may be used to train its generative AI foundation models. OpenAI documents that it 'is used to crawl content that may be used in training our generative AI foundation models'. It fetches on a schedule of OpenAI's own choosing, answers no user question, and serves no live response. It launched in August 2023 as the first mainstream training crawler with a published opt-out token.
If I block GPTBot, will I disappear from ChatGPT?
No. This is the most expensive misconception about the bot. GPTBot participates in neither ChatGPT Search nor live browsing — search inclusion is governed by OAI-SearchBot, and live fetches on a user's behalf come from ChatGPT-User. Blocking GPTBot keeps your content out of future training runs while leaving ChatGPT citation fully intact.
Does disallowing GPTBot remove my content from models already trained?
No. A robots.txt disallow governs future crawling only. It does not remove content from models that have already been trained, and OpenAI publishes no mechanism that does. Media Manager, the tool announced in May 2024 to let creators specify how their works are used, has not shipped.
Is there a GEO reason to allow GPTBot?
There is a plausible mechanism and no published evidence. Content absorbed during training can contribute to a model's prior about an entity, which is the same currency unlinked brand mentions trade in. But no study isolates 'this site allowed GPTBot' as a variable and shows a citation or mention lift, and the counterfactual is unavailable by construction — you cannot A/B a training run or attribute an unsourced mention to a crawl.
Does blocking GPTBot also block OpenAI's search crawler?
Not automatically — they are separate tokens with separate rules. But the two share infrastructure: 6 of GPTBot's 21 published IP prefixes also appear in OAI-SearchBot's list, and OpenAI states that if a site allows both bots, 'we may use the results from just one crawl for both use cases to avoid duplicative crawling'. The access decision is still made per token in robots.txt.

See also

Sources

Primary

  1. Overview of OpenAI Crawlers · OpenAI
  2. GPTBot published IP ranges (gptbot.json) · OpenAI
  3. OAI-SearchBot published IP ranges (searchbot.json) · OpenAI
  4. ChatGPT-User published IP ranges (chatgpt-user.json) · OpenAI
  5. RFC 9309: Robots Exclusion Protocol · IETF · 2022-09-01
  6. Your site, your rules: new AI traffic options for all customers · Cloudflare · 2026-07-01
  7. Control content use for AI training with Cloudflare's managed robots.txt · Cloudflare · 2025-07-01
  8. Source-Aware Training Enables Knowledge Attribution in Language Models · Khalifa et al. (arXiv) · 2024-04-01
  9. How many news websites block AI crawlers? · Reuters Institute for the Study of Journalism

Secondary

  1. A deeper look at AI crawlers: breaking down traffic by purpose and industry · Cloudflare
  2. The 2025 Cloudflare Radar Year in Review · Cloudflare
  3. OpenAI says it's building a tool to let content creators opt out of AI training · TechCrunch
  4. OpenAI failed to deliver the opt-out tool it promised by 2025 · TechCrunch
Last updated: 2026-07-28 Authors: Ray Yang Topic: Infrastructure