Free AI Crawler Access Checker

Enter a domain and see how it treats 26 AI crawler tokens: what its robots.txt says, what its edge actually serves, and what each class of block costs you.

Free, no account needed — 5 checks a day. Sign in for 15.

One robots.txt fetch, evaluated for every token per RFC 9309, each verdict quoting the exact line it came from. Then one probe per bot user agent against the same page, compared with a browser baseline to surface UA-keyed edge rules.

Updated 18 August 2026

Roster verified against operator docs 2026-08-18

Training crawlers

Blocking these costs future parametric “memory” of your content in model weights — a delayed, unattributable effect. It does not remove you from AI answers: citation is carried by the retrieval class below.

Crawler	robots.txt	Edge response
`GPTBot` · OpenAI operator docs	—	—
`ClaudeBot` · Anthropic operator docs	—	—
`Google-Extended` · Google robots.txt token only operator docs	—	Not probed — robots.txt token only
`Applebot-Extended` · Apple robots.txt token only operator docs	—	Not probed — robots.txt token only
`CCBot` · Common Crawl operator docs	—	—
`meta-externalagent` · Meta operator docs	—	—
`Amazonbot` · Amazon operator docs	—	—
`MistralAI-Training` · Mistral AI operator docs	—	—
`AI2Bot` · Allen Institute for AI operator docs	—	—
`Bytespider` · ByteDance observed ignoring robots.txt no operator docs · reported by blog.cloudflare.com	—	—

Retrieval / search crawlers

Blocking these costs citation now — you disappear from AI answers immediately. Googlebot, bingbot and Applebot also carry classic search, so blocking them exits Google, Bing or Siri/Spotlight entirely, not just their AI features.

Crawler	robots.txt	Edge response
`OAI-SearchBot` · OpenAI operator docs	—	—
`Claude-SearchBot` · Anthropic operator docs	—	—
`PerplexityBot` · Perplexity operator docs	—	—
`Googlebot` · Google also carries classic search operator docs	—	—
`bingbot` · Microsoft also carries classic search operator docs	—	—
`Applebot` · Apple also carries classic search operator docs	—	—
`DuckAssistBot` · DuckDuckGo operator docs	—	—
`YouBot` · You.com operator docs	—	—
`MistralAI-Index` · Mistral AI operator docs	—	—

User-triggered agents

Blocking these makes one user’s live lookup of your page fail mid-answer. Nothing automated is stopped: these fetch only when a human asks about your URL — and several document that they may ignore robots.txt anyway.

Crawler	robots.txt	Edge response
`ChatGPT-User` · OpenAI may ignore robots.txt (own docs) operator docs	—	—
`Claude-User` · Anthropic operator docs	—	—
`Perplexity-User` · Perplexity may ignore robots.txt (own docs) operator docs	—	—
`meta-externalfetcher` · Meta may ignore robots.txt (own docs) operator docs	—	—
`MistralAI-User` · Mistral AI operator docs	—	—
`Google-GeminiNotebook` · Google may ignore robots.txt (own docs) operator docs	—	—
`Google-Agent` · Google may ignore robots.txt (own docs) operator docs	—	—

How the check works

Fetch the site’s robots.txt once and parse it per RFC 9309: group merging, longest-token agent matching, longest-pattern rule precedence, * and $ wildcards.
Evaluate the homepage path for all 26 tokens and quote the exact User-agent group and rule line behind every verdict.
Fetch the homepage once with a normal browser user agent — the baseline every probe is compared against.
Send one request per probeable bot UA (24 of 26 — Google-Extended and Applebot-Extended exist only as robots.txt tokens) and read nothing but the status line and headers.
Report differences: a bot UA answered with 403/429 or an interactive challenge where the browser got the page reveals a UA-keyed edge rule. If the site blocks even the browser baseline, probes are reported as inconclusive instead of guessed.

What this check can and cannot see

A user-agent string is a claim, not an identity. Probes present each bot’s UA from our server’s IP addresses, so an edge that verifies crawlers by IP ranges or reverse DNS may treat them as impostors. An edge “blocked” verdict means a UA-keyed rule exists — not that the real crawler is blocked. The verification mechanics live in AI crawlers.
An identical response for bot and browser means no UA-level rule was observed — not that the real crawler gets in. IP-level and verified-bot rules are invisible to this test.
robots.txt is a voluntary request, not access control (RFC 9309). The roster marks agents whose own operators say user-triggered fetches may ignore it, and one crawler measured ignoring it.
Verdicts cover the homepage path. A site can allow the homepage and still disallow the paths that matter — rows carrying path-specific rules are marked “some paths restricted”.

Frequently asked questions

Should I block AI training crawlers?

That is a policy choice, and it is the one block with no immediate visibility cost: a training block keeps your content out of future model weights while retrieval crawlers keep citing you. The costly mistake is the opposite one — blanket-blocking every AI bot and silently removing yourself from AI answers. The category asymmetry is laid out in AI crawlers.

robots.txt says blocked but the edge probe says served — which is right?

Both. They measure different layers: robots.txt is your declared policy, which compliant crawlers check before fetching; the edge probe measures what the server enforces. A compliant crawler that reads a Disallow never sends the request, so the edge has nothing to block.

The edge probe says blocked — is the real crawler actually blocked?

Not necessarily. Our probe carries the bot’s user-agent string but comes from our IP addresses, so a WAF that verifies crawler identity by IP or reverse DNS sees an impostor and may block it while admitting the genuine bot. Read an edge “blocked” as “a UA-keyed rule exists at the edge”, then confirm against your own server logs.

Why are Google-Extended and Applebot-Extended never probed?

Neither sends a single HTTP request. Both are robots.txt-only control tokens: Googlebot and Applebot do the fetching, and these tokens only govern whether what was fetched may train models. Probing a UA that does not exist on the wire would measure nothing, so the tool evaluates them in robots.txt only.

Does blocking Googlebot or bingbot only affect AI features?

No — that is the most expensive block on the page. Google’s AI Overviews and AI Mode ride on Googlebot’s ordinary crawl, and Microsoft documents no separate Copilot crawler, so blocking either token removes you from classic Google or Bing search as well. Training use is controlled separately: Google-Extended for Google, content controls rather than a crawler block for Bing.