Skip to content
Intermediate Time: 12 min
Step 2 of 8

Crawler profiles: GPTBot, ClaudeBot, and PerplexityBot

Vendor names alone do not tell you what a crawler does or which policy applies. Compare the documented identifiers and purposes with what appears in your own logs, then investigate the crawlers that account for the most consequential traffic first.

Core reading

  1. GPTBot
    Wiki
  2. ClaudeBot
    Wiki · coming soon
  3. PerplexityBot
    Wiki · coming soon

Self-check

  • What are each crawler's User-Agent, documented purpose, observed behavior, and opt-out mechanism?
  • Which crawlers serve both model-training and live-retrieval roles, and how does that affect access policy?
  • Which of these is actually showing up in your traffic logs today?