AI Crawler Access Audit

1. What this audit answers

An AI crawler access audit verifies that the crawlers you decided to allow can actually fetch your pages, and that the ones you decided to block actually stopped. Both halves need evidence. Reading your own robots.txt back to yourself establishes neither.

The reason is that robots.txt is a declaration, not a delivery. A site can explicitly allow a retrieval crawler in the file while a CDN bot-management default, a WAF managed rule, a geo-block, or a rate limit stops that crawler at the network layer. The file will never show it. This is the most expensive class of access failure precisely because the artifact everyone checks is the one place it cannot appear.

The other direction needs evidence too. Writing Disallow does not prove anyone stopped — RFC 9309 states plainly that its rules “are not a form of access authorization,” and compliance is a published policy rather than a technical guarantee. Who actually arrived is a question only logs answer, and only after identity is confirmed.

So the output is a divergence list: each finding names two states that disagree, the evidence for the disagreement, a severity, and the layer that has to fix it — content, configuration, or network. The four-state split used below is GEO Wiki’s organizing device for that reconciliation, not an established audit standard; it earns its place because the divergences, not the states, are what turn out to be actionable.

2. Before you audit — scope, bot set, intended policy, evidence

Four decisions fix what every later finding means.

The intended policy row carries more weight than its size suggests. Every finding in the report is a deviation from intent, not a deviation from maximum openness. A site that deliberately declines training crawlers has not produced a finding when GPTBot is disallowed — that is the policy working. A retrieval crawler stopped by a WAF under that same policy is a finding, because nobody chose it.

When to run. After any CDN or WAF change, a migration, a new security layer, a CMS or plugin swap, when a new bot ships — and on vendor default-policy change dates, one of which is dated in §7.

3. The four states of access

Four states, each with its own evidence source.

None of the four is interesting on its own. What produces an actionable finding is the difference between two adjacent states, and there are only three such differences:

Work the states in order — declared, effective, observed — because each is more expensive than the last, and cheap evidence often resolves an expensive question before you go looking for it.

4. Step 1 — Audit the declared layer

Fetch the file rather than looking at it in a browser, and fetch it from every host in scope:

for h in example.com www.example.com docs.example.com blog.example.com; do
  printf '%-24s ' "$h"
  curl -sS -o "robots-$h.txt" -w '%{http_code} %{size_download}B\n' "https://$h/robots.txt"
done
shasum robots-*.txt

Two verdicts come out of this. A 4xx means no rules are declared at all for that host — a compliant crawler is then free to fetch any path. And identical hashes mean one policy is genuinely in force everywhere, while differing hashes mean you have per-host policies that someone needs to have chosen deliberately. Grammar, group selection, and path precedence follow the ordinary rules described in robots.txt.

Check for an edge override. A CDN can inject or replace robots.txt at the edge, which means the file you serve and the file in your repository can differ without anyone editing either. The test is a content comparison — edge-fetched bytes against repository bytes. That is more reliable than any configuration panel, because it measures the artifact instead of the intent behind it.

Page-level directives belong to this layer too. X-Robots-Tag response headers and robots meta tags are declared policy, but they control indexing and display rather than fetching:

curl -sSI "https://example.com/pricing" | grep -i 'x-robots-tag'

The distinction matters when reconciling later. Google’s documentation states that if a page is blocked in robots.txt, “Googlebot will never crawl the page and will never read any meta tags on the page” (robots meta tag specifications). A crawl block and an index block produce different symptoms and have different fixes; recording which one is in play here saves a misdiagnosis in §8.

One note on llms.txt: it grants no access and denies none, so it produces no finding in this audit. Whether it is present and accurate is a question for llms.txt deployment.

5. Step 2 — Probe the effective layer

Request the same URL twice — once as a baseline, once as the bot — and compare three things, not one:

URL="https://example.com/pricing"
UA_BROWSER="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
UA_BOT="PerplexityBot"   # the bare product token is enough for most UA rules

curl -sS -A "$UA_BROWSER" -o base.html -w 'baseline %{http_code} %{size_download}B\n' "$URL"
curl -sS -A "$UA_BOT"     -o bot.html  -w 'bot      %{http_code} %{size_download}B\n' "$URL"
shasum base.html bot.html

Status code alone is not enough, which is why size and hash are in the command. A challenge page and a soft block both return 200. The full published user-agent strings, which carry version numbers that change, are listed per bot in GPTBot, ClaudeBot, and PerplexityBot.

Run this across a matrix rather than one URL: the homepage, one core content template, one paginated or filtered path, and one restricted path. Record URL template × bot token × status × bytes × verdict.

Delivered but empty is the one finding this audit detects and hands off. A page that returns 200 with the real bytes missing because the content requires client-side rendering is a Layer-2 problem in the full GEO audit ladder, and the fix is described in SSR for AI crawlers. Detect it here, because the body-size comparison you are already running catches it for free; do not try to fix it here.

The method limit, stated plainly. A spoofed-user-agent probe can prove permissiveness and cannot prove blocking. Vendor crawlers are typically allowlisted by published IP range or reverse DNS rather than by the user-agent string — Cloudflare’s verified-bot program, for instance, requires honest self-identification through a Web Bot Auth signature, a published IP list, or reverse DNS validation, alongside non-abusive behavior (verified bots). So a 403 returned to your fake GPTBot most likely means anti-spoofing is working correctly, not that the real crawler is blocked. The inference runs one way: a 200 to the fake token implies the real one gets through. Confirming a block requires §6.

6. Step 3 — Verify the observed layer

Not everyone has an origin access log, and assuming otherwise strands the readers who most need this step.

The critical property: an origin log cannot see a request the CDN stopped at the edge. The Declared ≠ Effective divergence from §3 leaves its trace only in edge-side data, which is exactly why that class goes unnoticed on sites that check origin logs and stop.

Retention is the other trap, and it is often shorter than people assume. Vercel documents runtime-log retention of 1 hour on Hobby and 1 day on Pro (30 days with Observability Plus), and shows static requests in runtime logs only when they serve cache — full static logging requires log drains (Vercel runtime logs). A crawler that revisits monthly is invisible in a one-day window.

Summarize arrivals per token:

# Which AI tokens arrived, from which IPs, and what did they get?
# Adjust awk fields to your log format ($1 = client IP, $9 = status).
grep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot|Bingbot' access.log \
  | awk '{print $1, $9}' | sort | uniq -c | sort -rn | head -40

What you want out of it is a status distribution per token, not a total. A token arriving steadily with a wall of 403s is a confirmed block. A token arriving with 200s is confirmed access — once its identity holds.

Identity is two steps, not one. Compare the source IP against the operator’s published list, then run forward-confirmed reverse DNS. Google documents the sequence directly: reverse-lookup the accessing IP, then forward-lookup the returned hostname and confirm it resolves back (verify Google requests).

host 203.0.113.10          # → PTR hostname
host crawl-203-0-113-10.googlebot.com   # → must resolve back to 203.0.113.10

Published endpoints, verified 29 July 2026:

Two structural facts change how you read this table. Google-Extended is a control token with no crawler of its own, so it generates no log lines at all — the observed layer has nothing to say about it by construction, and its effect can only be checked in the declared layer. And published IP lists confirm identity on inbound requests; they are not durable block lists. Anthropic says so directly: “alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out,” because its crawlers run on public cloud addresses (Anthropic crawler documentation).

Reading an absence correctly. No log entries does not mean blocked. The crawler may not have discovered the pages, may have deprioritized them, may revisit less often than your retention window, or may not use its own crawler for pages like yours. Cross-read it against §5: a clean effective layer with an empty observed layer is a discovery and priority question, answered by strengthening the change and coverage signals described in sitemap and IndexNow, not by touching access at all.

One category resists this method. User-triggered fetches happen when a person asks about a specific URL, so their volume is small and irregular, and arrival counts cannot validate a policy either way. Whether robots.txt binds them at all is a separate documented question, treated in ChatGPT-User.

7. Where the infrastructure layer overrides you

An increasing share of what actually happens is decided somewhere other than your files.

Detection quality varies by tier, which affects what your own controls actually do. Cloudflare documents that “on the free plan, AI Crawl Control identifies AI crawlers based on their user agent strings,” with more thorough Bot Management detection available on paid plans, and that blocking can return either 403 or 402 (manage AI crawlers). A user-agent-based control and an identity-based control fail differently, and knowing which one you have changes how you read a 403 in §5.

One dated example is worth planning around. From 15 September 2026, Cloudflare states that “Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default,” landing on newly onboarded domains with existing customers able to set a preference beforehand (Cloudflare). Whatever the merits, the audit consequence is structural: effective access changes with no edit to any file you own.

The direction, stated no further than the evidence allows: control is migrating from a text file toward the network layer and toward contracts. Cryptographic bot authentication is the plausible endpoint — the IETF has chartered a Web Bot Auth working group, and draft-meunier-web-bot-auth-architecture proposes letting automated clients “cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence.” That draft is an individual submission listed as a possible input to the working group, not a standard, so treat it as a direction rather than a control you can audit against today. For the audit, the practical consequence is that declared-layer evidence is losing value while effective and observed evidence gains it.

8. Reconcile — classify findings and prioritize

Every finding is written as a named divergence plus its evidence plus the layer that owns the fix. Severity uses the same vocabulary as the full GEO audit, but anchors to the crawler category affected rather than to a layer number.

Severity is not priority. Adding one WAF allowlist entry and migrating off a client-rendered architecture are not comparable pieces of work, so re-rank the severity-tagged list by impact, confidence, and ease before it becomes an action plan — the same reordering the full GEO audit applies across all six of its layers.

9. Validity threats and pitfalls

• Probing from inside your own network. Office IPs are frequently allowlisted; the result does not generalize to the public internet.
• Reading a 403 to a fake user agent as proof of a block. The single most expensive misreading here — see §5.
• Comparing status codes only. Misses challenge pages and empty 200 shells entirely.
• Checking origin logs alone. Edge-blocked requests never reach the origin, and that is the class you are hunting.
• Auditing the apex domain only. robots.txt is host-scoped; every subdomain needs its own check.
• Log retention shorter than the crawl interval. The observed layer then cannot support any conclusion.
• Sampling a single URL template. Path-scoped rules and per-template differences average out of the result.
• Re-checking immediately after a robots.txt edit. Propagation is not instant, and published intervals differ by operator and often cover only the search side — the documented figure for OpenAI is scoped to search results, as recorded in GPTBot.
• Treating “not cited” as an access problem. That is a different instrument entirely; see AI citation tracking.
• Auditing once. Vendor defaults and managed rule sets change the answer without your involvement (§7).

10. The report deliverable and re-check cadence

What ships, every time:

• Header — audit date, the list of hosts in scope, the bot set, the version of the intended policy audited against, and the evidence window.
• Four-state record — what each state was, per host, with the command output or log excerpt behind it.
• Divergence list — each finding named as a state pair, with severity and owning layer.
• Prioritized fixes — the re-ranked list from §8.
• Delta — what changed since the last audit, and whether it changed because you acted or because a vendor did.

Split the cadence by cost. The declared and effective layers are cheap enough to run on a schedule, which is what catches a vendor default flipping or a managed rule set updating. The observed layer travels with the full audit — quarterly, or on the triggers in §2.

Then write the review date into the policy itself, next to the directives. The restriction landscape moves on its own: a longitudinal audit of 14,000 web domains found that within a single year, 2023 to 2024, more than 5% of all tokens in the C4 corpus — and over 28% of its most actively maintained sources — became fully restricted (Longpre et al., 2024). A policy set once and never revisited is a policy aimed at a web that has moved.

11. Related entries

• AI crawlers — the three categories and the per-category access decision
• robots.txt — the protocol, group selection, and how directives are parsed
• GPTBot · ClaudeBot · PerplexityBot — per-bot user agents, IP files, and block recipes
• OAI-SearchBot — the token that actually governs ChatGPT search inclusion
• ChatGPT-User — user-triggered fetching and whether robots.txt binds it
• Google-Extended — a control token with no crawler, and therefore no log evidence
• SSR for AI crawlers — fixing a 200 that arrives empty
• llms.txt — what it is, and why it grants no access
• Full GEO audit — the six-layer ladder this audit sits at the bottom of
• AI citation tracking — the instrument for “reachable but not cited”

References

Primary

• IETF — RFC 9309: Robots Exclusion Protocol
• IETF — HTTP Message Signatures for automated traffic: Architecture (Internet-Draft, version 05, 2 March 2026; not a standard)
• Google Search Central — Verify requests from Google crawlers and fetchers · common-crawlers.json · New location for the crawler IP range files · Robots meta tag and X-Robots-Tag specifications
• OpenAI — Overview of OpenAI Crawlers
• Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler? · crawler IP list
• Perplexity — Perplexity Crawlers · perplexitybot.json · perplexity-user.json
• Microsoft Bing — How to verify Bingbot · Verify Bingbot tool
• Cloudflare — Verified bots · Manage AI crawlers · Your site, your rules: new AI traffic options (1 July 2026)
• Vercel — Runtime logs and retention limits

Secondary

• Longpre, S. et al. — Consent in Crisis: The Rapid Decline of the AI Data Commons (arXiv:2407.14933, July 2024)
• Search Engine Land — Anthropic clarifies how Claude bots crawl sites and how to block them
• Cloudflare — Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (4 August 2025)