AI Crawlers

1. What an AI crawler is

An AI crawler is an automated agent that fetches pages on behalf of an AI system. Its role matters more than the user-agent name it presents.

Definition (GEO Wiki working definition): An AI crawler is an automated agent that fetches pages on behalf of an AI system. In GEO, the term covers the three roles that affect AI visibility: training, retrieval, and user-triggered access. It does not refer to every nonhuman user agent.

Exact user-agent strings, published IP ranges, and blocking instructions differ among GPTBot, ClaudeBot, and PerplexityBot.

Managing crawler access involves three separate questions:

• Which protocols express access preferences? See robots.txt and llms.txt.
• How should a team write the policy and audit its logs? Follow the AI Crawler Access Audit playbook.
• Can the crawler read the page after it arrives? See SSR for AI Crawlers.

An AI Crawler Access Audit combines the declared policy with log evidence so teams can compare crawler rules with actual traffic.

2. The three crawler categories

Training crawlers, retrieval crawlers, and user-triggered agents serve different purposes. Blocking each category therefore has different consequences. Treating all three as a single class can unintentionally restrict a site’s current visibility in AI answers.

                         ┌──────────────► TRAINING crawler
                         │                 (GPTBot, ClaudeBot,
                         │                  Google-Extended)
                         │                 → feeds/refines model
                         │                   weights; delayed,
                         │                   parametric, not
   one origin page ──────┤                   attributable
   fetched for           │
   three reasons         ├──────────────► RETRIEVAL crawler
                         │                 (OAI-SearchBot,
                         │                  PerplexityBot,
                         │                  Claude-SearchBot)
                         │                 → grounds a live answer
                         │                   + builds the answer
                         │                   index; immediate,
                         │                   citation-bearing
                         │
                         └──────────────► USER-TRIGGERED agent
                                           (ChatGPT-User,
                                            Claude-User,
                                            Perplexity-User)
                                          → one user asked about
                                            your URL right now

Access decisions should be made by category, not by individual bot or through one binary rule. Blocking a training crawler does not block the corresponding retrieval crawler. Blocking a retrieval crawler removes a site from current AI answers.

In step 2 of the answer loop, crawler access determines whether a page can become a candidate source. Citability then determines whether an engine can extract a useful passage after fetching it. The remaining stages of the answer loop occur only after retrieval.

3. How categories map to crawlers

The table groups representative crawlers by purpose and shows what a site gains or loses by blocking each category. Individual user-agent strings, IP-verification methods, and blocking instructions vary by crawler. Crawler lists change quickly, so verify the current list against primary documentation before changing access rules.

The consequences are asymmetric. Blocking training gives up future parametric memory while preserving citations. Blocking retrieval gives up current citations even though training has already occurred or continues through other corpora. The two controls are not interchangeable.

Different product surfaces depend on different crawlers. ChatGPT search uses OAI-SearchBot and ChatGPT-User. Perplexity uses PerplexityBot and Perplexity-User. Claude uses Claude-SearchBot and Claude-User. Google AI Overviews relies on Googlebot; Google-Extended governs training use rather than crawling (see Google’s common crawlers). Bing Copilot relies on Bingbot. Exact user-agent strings and IP-verification details vary by crawler.

4. User-agent is a claim, not an identity

A user-agent string is self-asserted and easy to spoof. Seeing GPTBot in a log does not prove that OpenAI fetched the page. A policy based only on that string can block legitimate bots that have been impersonated while failing to stop malicious agents using a trusted name.

Identity is established through published IP ranges and forward-confirmed reverse DNS (rDNS). Major operators provide machine-readable IP lists for this purpose. OpenAI publishes separate .json files for its crawlers (see Overview of OpenAI Crawlers). Anthropic publishes a list of bot IPs (see Anthropic crawler docs). Perplexity publishes separate IP JSON files for its crawlers (see Perplexity Crawlers).

The following generic procedure shows the principle behind forward-confirmed rDNS verification:

# Forward-confirmed rDNS verification principle
1. reverse-lookup the request IP        → host name
2. forward-lookup that host name        → IP
3. step-2 IP == request IP  AND  host in the operator's domain
   → identity confirmed; else → treat the UA as unverified

Exact IP ranges, rDNS domains, and runnable verification steps vary by crawler. The AI Crawler Access Audit playbook provides the operational procedure. Cryptographic bot authentication is an emerging alternative. The proposed HTTP Message Signatures architecture for automated traffic (IETF draft-meunier-web-bot-auth-architecture) would replace IP and user-agent inference with signed identities. It remains an Internet-Draft with no formal IETF standing, so it cannot yet serve as a dependable control.

5. What the evidence shows: declaring a rule does not enforce it

robots.txt is a voluntary request, not an access-control mechanism. RFC 9309 states that its rules “are not a form of access authorization” and that the protocol “is not a substitute for valid content security measures” (see RFC 9309). Without retrieval, citability cannot affect whether a page appears in an answer.

The PerplexityBot case illustrates the documented gap between declaring a rule and enforcing it. In 2024, multiple news outlets reported that a major answer engine fetched content from sites that had disallowed its crawler (TechCrunch, 2024-07-02). In 2025, Cloudflare reported that an undeclared crawler impersonated a normal browser and rotated IPs to evade no-crawl directives across tens of thousands of domains (Cloudflare, 2025-08-04).

Use robots.txt to state the policy because compliant crawlers follow it. Then use the AI Crawler Access Audit to verify which agents actually reached the site. A declared rule does not by itself provide enforcement.

6. When blocking AI crawlers backfires

These common approaches fail because they confuse crawler categories, policy declarations, and technical enforcement.

Blocking every AI crawler is not the safest default. It is the costliest one. Each block trades away the benefits of a specific crawler category, including citations. OpenAI states that appearing and being cited in ChatGPT search requires allowing its search crawler (see Publishers and Developers FAQ).

7. How AI crawling differs from traditional SEO crawling

AI crawlers inherit the technical baseline described in SEO vs GEO, but they introduce a different set of access decisions.

The technical baseline is unchanged: pages must be reachable, return a 200 status, avoid soft 404s, and maintain a reasonable crawl budget and clean status codes. These are the same conditions Googlebot has always required.

The traditional SEO model assumes one main crawler, one access decision, and reliable compliance. AI crawling involves many agents, three functional categories, and less reliable compliance.

8. What crawler access means for GEO

Reachability is necessary but not sufficient. It makes a page a candidate source. Citability then determines whether an engine can extract and attribute a useful passage from that page.

A sound access strategy requires three separate decisions: choosing access by crawler category, verifying identity at the network layer, and keeping pages readable after retrieval. Each decision addresses a different failure mode, and all three affect whether a site can appear in AI answers.

References

Official crawler documentation (as of 2026-05):

• OpenAI: Overview of OpenAI Crawlers · Publishers and Developers FAQ
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity: Perplexity Crawlers
• Google Search Central: Overview of Google crawlers and fetchers · Google’s common crawlers (Google-Extended)
• Microsoft Bing: Which crawlers does Bing use?

Protocol and standards:

• IETF: RFC 9309: Robots Exclusion Protocol
• IETF: HTTP Message Signatures for automated traffic: Architecture (draft-meunier-web-bot-auth-architecture) (Internet-Draft; emerging, not yet a standard)

Independent measurement and reporting:

• Cloudflare: From Googlebot to GPTBot: who’s crawling your site in 2025
• Cloudflare: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
• TechCrunch: News outlets are accusing Perplexity of plagiarism and unethical web scraping
• Search Engine Land: Anthropic’s Claude bots make robots.txt decisions more granular