Free AI Crawler Access Checker

Enter a domain to see what its robots.txt allows, how its edge responds to AI crawler user agents, and which types of access may be affected.

Free, no account required. Run 5 checks per day, or sign in for 15.

The checker reads robots.txt once and evaluates every token under RFC 9309, showing the rule behind each result. It then compares selected crawler user agents with a browser baseline to identify possible CDN or WAF differences.

Crawler list verified against operator documentation on 2026-08-18

Training crawlers

Blocking these crawlers may prevent your content from being used in future model training. It does not by itself control whether retrieval systems can find or cite your pages.

Crawler	robots.txt	Edge response
`GPTBot` · OpenAI operator docs	—	—
`ClaudeBot` · Anthropic operator docs	—	—
`Google-Extended` · Google robots.txt token only operator docs	—	Not tested; robots.txt token only
`Applebot-Extended` · Apple robots.txt token only operator docs	—	Not tested; robots.txt token only
`CCBot` · Common Crawl operator docs	—	—
`meta-externalagent` · Meta operator docs	—	—
`Amazonbot` · Amazon operator docs	—	—
`MistralAI-Training` · Mistral AI operator docs	—	—
`AI2Bot` · Allen Institute for AI operator docs	—	—
`Bytespider` · ByteDance observed not following robots.txt no operator docs · reported by blog.cloudflare.com	—	—

Retrieval / search crawlers

Blocking these crawlers can reduce visibility in AI search and citations. Googlebot, bingbot, and Applebot also support traditional search or discovery, so restrictions may have effects beyond AI features.

Crawler	robots.txt	Edge response
`OAI-SearchBot` · OpenAI operator docs	—	—
`Claude-SearchBot` · Anthropic operator docs	—	—
`PerplexityBot` · Perplexity operator docs	—	—
`Googlebot` · Google also used for traditional search operator docs	—	—
`bingbot` · Microsoft also used for traditional search operator docs	—	—
`Applebot` · Apple also used for traditional search operator docs	—	—
`DuckAssistBot` · DuckDuckGo operator docs	—	—
`YouBot` · You.com operator docs	—	—
`MistralAI-Index` · Mistral AI operator docs	—	—

User-triggered agents

Blocking these agents may prevent a product from opening your page when a user asks for it. Some operators document different robots.txt behavior for user-requested fetches, so review the notes for each agent.

Crawler	robots.txt	Edge response
`ChatGPT-User` · OpenAI operator notes different robots.txt behavior operator docs	—	—
`Claude-User` · Anthropic operator docs	—	—
`Perplexity-User` · Perplexity operator notes different robots.txt behavior operator docs	—	—
`meta-externalfetcher` · Meta operator notes different robots.txt behavior operator docs	—	—
`MistralAI-User` · Mistral AI operator docs	—	—
`Google-GeminiNotebook` · Google operator notes different robots.txt behavior operator docs	—	—
`Google-Agent` · Google operator notes different robots.txt behavior operator docs	—	—

How the check works

Fetch and parse robots.txt according to RFC 9309, including group merging, user-agent matching, rule precedence, and * or $ patterns.
Evaluate the homepage path for all 26 tokens and show the User-agent group and rule behind each result.
Request the homepage with a standard browser user agent to establish a comparison baseline.
Send one lightweight request for each crawler user agent that can be tested. Google-Extended and Applebot-Extended are evaluated only as robots.txt tokens.
Compare the responses. If a crawler user agent receives a block, challenge, or different redirect while the browser succeeds, report a possible user-agent-specific edge rule. If the browser baseline is also blocked, mark the result as inconclusive.

What this check can and cannot see

A user-agent string does not prove crawler identity. These probes come from our server IP addresses, so systems that verify crawlers by IP range or reverse DNS may treat them differently from the genuine crawler. An edge block means the probe was blocked, not necessarily the real crawler. See AI crawlers for verification methods.
Matching browser and crawler responses mean no user-agent-specific difference was observed. This test cannot see restrictions based only on IP address or verified-bot status.
robots.txt is a published crawling preference, not access control. Some user-triggered agents document different behavior, and the table marks them accordingly.
Results are based on the homepage path. Other paths may have different rules; those rows are marked “some paths restricted”.

Frequently asked questions

Should I block AI training crawlers?

It depends on your content policy. Training crawlers and retrieval crawlers serve different purposes, so you can restrict training access without automatically blocking AI search access. Review each crawler rather than applying one rule to every AI-related user agent. The categories are explained in AI crawlers.

robots.txt says blocked, but the edge probe says served. Which is correct?

Both can be correct because they measure different layers. robots.txt publishes your crawling policy, while the edge probe tests what the server returns when a request is made. A crawler that follows a Disallow rule may never send that request.

The edge probe says blocked. Is the real crawler also blocked?

Not necessarily. The probe uses the crawler’s user-agent string but comes from our IP addresses. A WAF that verifies crawler identity by IP or reverse DNS may block the probe while allowing the genuine crawler. Confirm the result with your own server or CDN logs.

Why aren’t Google-Extended and Applebot-Extended tested at the edge?

They are control tokens used in robots.txt, not crawler user-agent strings used for separate page requests. Googlebot and Applebot perform the fetching, so this tool evaluates the Extended tokens only in robots.txt.

Does blocking Googlebot or bingbot only affect AI features?

No. Googlebot and bingbot also support traditional search, so blocking them can affect more than AI features. Use separate training controls where available, and review the operator documentation before changing a broad rule.