AI Crawler Access Audit

1. What this audit answers

An AI crawler access audit verifies that crawlers you intended to allow can fetch your pages and that crawlers you intended to block have stopped. Both conclusions require evidence. Reading robots.txt alone establishes neither one.

robots.txt is a declaration, not evidence of delivery. A site can explicitly allow a retrieval crawler while a CDN bot-management default, WAF rule, geographic block, or rate limit stops it at the network layer. The file cannot reveal that failure. This is the most costly class of access problem because it does not appear in the file people are most likely to check.

The opposite conclusion also needs evidence. Writing Disallow does not prove that a crawler stopped. RFC 9309 states that its rules “are not a form of access authorization,” and compliance is a published policy rather than a technical guarantee. Only logs can show which crawlers arrived, and those records are trustworthy only after identity verification.

The resulting report is a divergence list. Each finding identifies two states that disagree, the supporting evidence, the severity, and the responsible layer: content, configuration, or network. The four-state model in this guide is a GEO Wiki framework rather than an established audit standard. Its purpose is to make the differences between states explicit, because those differences lead to specific actions.

2. Before you audit: Define the scope, bot set, intended policy, and evidence window

Make four decisions before collecting evidence. They determine how every later finding should be interpreted.

The intended policy provides the baseline for the entire report. A finding is a departure from that policy, not from maximum openness. If a site deliberately excludes training crawlers, disallowing GPTBot shows that the policy is working. If a WAF also stops a retrieval crawler that the policy allows, that block is a finding because it was not intended.

When to run the audit. Repeat it after a CDN or WAF change, a migration, the addition of a security layer, a CMS or plugin change, the release of a new bot, or a change to a vendor’s default policy. Section 7 includes a dated example of the last case.

3. The four states of access

Each of the four states answers a different question and requires its own evidence.

No single state can produce an actionable finding by itself. The finding comes from a difference between two adjacent states, and there are three possible differences:

Review the declared, effective, and observed states in that order. Investigating each successive state costs more than investigating the previous one, and inexpensive evidence may answer a question before you need to examine the next layer.

4. Step 1: Audit the declared layer

Fetch robots.txt directly from every host in scope instead of checking a single copy in a browser:

for h in example.com www.example.com docs.example.com blog.example.com; do
  printf '%-24s ' "$h"
  curl -sS -o "robots-$h.txt" -w '%{http_code} %{size_download}B\n' "https://$h/robots.txt"
done
shasum robots-*.txt

These checks produce two kinds of evidence. A 4xx means that the host declares no rules, so a compliant crawler may fetch any path. Identical hashes show that the same policy is active on each host, while different hashes show that the hosts apply different policies. Confirm that those differences are intentional. Grammar, group selection, and path precedence follow the rules described in robots.txt.

Check for an edge override. A CDN can inject or replace robots.txt at the edge. As a result, the served file may differ from the version in your repository even when nobody has edited either one. Compare the bytes fetched from the edge with the repository version. This measures the file visitors receive and is more reliable than a configuration-panel screenshot.

Include page-level directives. X-Robots-Tag response headers and robots meta tags are also declared policy, but they control indexing and display rather than fetching:

curl -sSI "https://example.com/pricing" | grep -i 'x-robots-tag'

Record the distinction now so you can interpret the evidence correctly in §8. Google’s documentation states that if a page is blocked in robots.txt, “Googlebot will never crawl the page and will never read any meta tags on the page” (robots meta tag specifications). Crawl blocks and index blocks produce different symptoms and require different fixes.

llms.txt neither grants nor denies access, so its presence does not produce a finding in this audit. Use llms.txt deployment to check whether that file is present and accurate.

5. Step 2: Probe the effective layer

Request the same URL twice: first with a browser user agent as the baseline, then with the target bot token. Compare three properties of the responses:

URL="https://example.com/pricing"
UA_BROWSER="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
UA_BOT="PerplexityBot"   # the bare product token is enough for most UA rules

curl -sS -A "$UA_BROWSER" -o base.html -w 'baseline %{http_code} %{size_download}B\n' "$URL"
curl -sS -A "$UA_BOT"     -o bot.html  -w 'bot      %{http_code} %{size_download}B\n' "$URL"
shasum base.html bot.html

A status code alone is not enough because challenge pages and soft blocks can both return 200. Response size and body hash reveal differences that the status code misses. For complete published user-agent strings, including version numbers that can change, see GPTBot, ClaudeBot, and PerplexityBot.

Run the comparison across a matrix of URLs rather than testing a single page. Include the homepage, a core content template, a paginated or filtered path, and a restricted path. For each test, record the URL template, bot token, status, response size, and verdict.

Delivered without primary content is the only result from these access checks that is categorized in another layer. It is a Layer-2 problem in the full GEO audit. It occurs when a page returns 200, but the primary content is absent because it requires client-side rendering. The response-size and body-hash comparison can detect this condition as part of the access test. See SSR for AI crawlers for the remediation.

Know the limit of this method. A spoofed-user-agent probe can demonstrate permissive access, but it cannot prove that a real crawler is blocked. Vendor crawlers are typically allowlisted by published IP range or reverse DNS rather than by the user-agent string. For example, Cloudflare’s verified-bot program requires honest self-identification through a Web Bot Auth signature, a published IP list, or reverse DNS validation, along with non-abusive behavior (verified bots). A 403 response to a fake GPTBot therefore most likely means that anti-spoofing works correctly. It does not show that the real crawler is blocked. The inference works in only one direction: a 200 response to the fake token implies that the real crawler can also get through. Use the log analysis in §6 to confirm a block.

The free AI Crawler Access Checker can automate the declared and effective checks in one pass. It evaluates robots.txt for 26 AI tokens, runs differential edge probes against a browser baseline, and quotes the rule that determined each verdict. It also applies the one-way inference described above. The log verification in §6 remains manual.

6. Step 3: Verify the observed layer

The available evidence depends on your hosting setup. Many sites do not have an origin access log, so begin with the log source you can actually inspect.

An origin log cannot show a request that the CDN stopped at the edge. A divergence between the declared and effective states appears only in edge data. Sites that review only origin logs will miss it.

Log retention is another common limitation. Vercel documents runtime-log retention of 1 hour on Hobby and 1 day on Pro, or 30 days with Observability Plus. It also shows static requests in runtime logs only when they are served from cache; full static logging requires log drains (Vercel runtime logs). A crawler that returns monthly will not appear in a one-day window.

Summarize requests by crawler token:

# Which AI tokens arrived, from which IPs, and what did they get?
# Adjust awk fields to your log format ($1 = client IP, $9 = status).
grep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot|Bingbot' access.log \
  | awk '{print $1, $9}' | sort | uniq -c | sort -rn | head -40

Calculate the status distribution for each token rather than reporting a single request total. Repeated requests with 403 responses confirm a block. Requests with 200 responses confirm access once you have verified the crawler’s identity.

Verify identity in two steps. Compare the source IP with the operator’s published list, then run forward-confirmed reverse DNS. Google documents the sequence directly: perform a reverse lookup on the source IP, then perform a forward lookup on the resulting hostname and confirm that it resolves to the original IP (verify Google requests).

host 203.0.113.10          # → PTR hostname
host crawl-203-0-113-10.googlebot.com   # → must resolve back to 203.0.113.10

The following published endpoints were verified on July 29, 2026:

Two facts affect how you interpret the table. Google-Extended is a control token rather than a crawler, so it never generates log entries. Its effect can be checked only in the declared layer. Published IP lists, meanwhile, confirm the identity of inbound requests but do not provide durable block lists. Because its crawlers use public cloud addresses, Anthropic states that “alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out” (Anthropic crawler documentation).

Interpret missing records carefully. An empty log does not show that a crawler was blocked. The crawler may not have discovered the pages or may have deprioritized them. Its revisit interval may exceed your log-retention window, or it may not use its own crawler for that type of page. Compare this absence with the probes in §5. If the effective layer returns a clean response but the observed layer remains empty, investigate discovery and priority. The change and coverage signals in sitemap and IndexNow address that problem; changing access controls does not.

User-triggered fetches require different interpretation. They occur when a person asks about a specific URL, so their volume is low and irregular. Arrival counts therefore cannot confirm whether a policy works. ChatGPT-User examines the separate question of whether robots.txt applies to these requests.

7. How infrastructure controls override site policy

Network and platform controls can change effective access without changing files in your repository.

Detection quality varies by service tier, so determine how the service identifies crawlers. Cloudflare documents that “on the free plan, AI Crawl Control identifies AI crawlers based on their user agent strings.” Paid plans can use more thorough Bot Management detection, and blocked requests may return either 403 or 402 (manage AI crawlers). User-agent-based and identity-based controls fail in different ways, which changes how you should interpret a 403 in §5.

One announced change illustrates the need for a dated review. Beginning September 15, 2026, Cloudflare states that “Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.” The setting applies to newly onboarded domains, while existing customers can choose a preference beforehand (Cloudflare). Regardless of the policy’s merits, it can change effective access without any edit to a file you control.

The evidence points to a shift from text-file controls toward network enforcement and contractual arrangements. Cryptographic bot authentication is one possible next step. The IETF has chartered a Web Bot Auth working group, and draft-meunier-web-bot-auth-architecture proposes allowing automated clients to “cryptographically sign outbound requests, allowing HTTP servers to verify their identity with confidence.” The draft is an individual submission listed as a possible input to the working group, not a standard or a control that can be audited today. The practical consequence is that evidence from the declared layer is becoming less informative, while evidence from the effective and observed layers is becoming more important.

8. Classify and prioritize the findings

Write each finding as a named divergence, followed by its evidence and the layer responsible for remediation. Use the severity terms from the full GEO audit, but base the rating on the crawler category affected rather than the layer number.

Severity and priority answer different questions. Adding a WAF allowlist entry and replacing a client-rendered architecture require very different levels of effort. Before turning the findings into an action plan, rank them by impact, confidence, and ease. The full GEO audit uses the same prioritization method across all six layers.

9. Validity threats and pitfalls

• Probing from inside your own network. Office IPs are often allowlisted, so the result may not represent access from the public internet.
• Treating a 403 response to a fake user agent as proof of a block. This is the most costly misinterpretation in the audit. See §5 for the limit of synthetic probes.
• Comparing only status codes. This misses challenge pages and empty 200 responses.
• Checking only origin logs. Requests blocked at the edge never reach the origin, so the evidence for that category of failure is absent.
• Auditing only the apex domain. Because robots.txt is scoped to a host, every subdomain requires a separate check.
• Using logs retained for less time than the crawler’s revisit interval. The observed layer cannot support a conclusion under those conditions.
• Sampling a single URL template. One sample can hide path-specific rules and differences among templates.
• Rechecking immediately after a robots.txt edit. Changes do not take effect instantly. Published intervals vary by operator and often apply only to search. See GPTBot for the documented OpenAI interval, which applies only to search results.
• Treating a lack of citations as an access problem. Investigate that outcome with AI citation tracking.
• Running the audit only once. Vendor defaults and managed rule sets can change access without your involvement (§7).

10. Prepare the report and set a recheck schedule

Include the following five parts in every report:

• Header. Record the audit date, hosts in scope, bot set, version of the intended policy, and evidence window.
• Four-state record. Document each state for every host and attach the relevant command output or log excerpt.
• Divergence list. Name the state pair for each finding, then record its severity and responsible layer.
• Prioritized fixes. Include the findings ranked according to §8.
• Changes since the previous audit. Explain what changed and whether you or the vendor caused the change.

Use a different review schedule for each type of evidence. The declared and effective layers are inexpensive enough for scheduled checks, which can detect changes to vendor defaults or managed rule sets. Review the observed layer during the full audit, either quarterly or after one of the events listed in §2.

Add the next review date to the written policy beside the directives. Restrictions change even when your policy does not. In a longitudinal audit of 14,000 web domains, more than 5% of all tokens in the C4 corpus and more than 28% of its most actively maintained sources became fully restricted between 2023 and 2024 (Longpre et al., 2024). A policy that is never reviewed will eventually describe conditions that no longer exist.

11. Related entries

• AI crawlers: the three crawler categories and the access decision for each one
• robots.txt: the protocol, group selection, and directive parsing
• GPTBot · ClaudeBot · PerplexityBot: user agents, IP files, and blocking instructions for individual crawlers
• OAI-SearchBot: the token that governs inclusion in ChatGPT search
• ChatGPT-User: user-triggered fetching and whether robots.txt applies to it
• Google-Extended: a control token that has no crawler and therefore produces no log evidence
• SSR for AI crawlers: remediation for a 200 response that contains no primary content
• llms.txt: what the file does and why it grants no access
• Full GEO audit: the six-layer framework in which access is the first layer
• AI citation tracking: the method for investigating pages that are reachable but not cited

References

Primary

• IETF: RFC 9309: Robots Exclusion Protocol
• IETF: HTTP Message Signatures for automated traffic: Architecture (Internet-Draft, version 05, March 2, 2026; not a standard)
• Google Search Central: Verify requests from Google crawlers and fetchers · common-crawlers.json · New location for the crawler IP range files · Robots meta tag and X-Robots-Tag specifications
• OpenAI: Overview of OpenAI Crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? · crawler IP list
• Perplexity: Perplexity Crawlers · perplexitybot.json · perplexity-user.json
• Microsoft Bing: How to verify Bingbot · Verify Bingbot tool
• Cloudflare: Verified bots · Manage AI crawlers · Your site, your rules: new AI traffic options (July 1, 2026)
• Vercel: Runtime logs and retention limits

Secondary

• Longpre, S. et al.: Consent in Crisis: The Rapid Decline of the AI Data Commons (arXiv:2407.14933, July 2024)
• Search Engine Land: Anthropic clarifies how Claude bots crawl sites and how to block them
• Cloudflare: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (August 4, 2025)