Skip to content
Technical GEO Intermediate
Step 1 of 8

The AI crawler landscape

Before you touch a single robots.txt line, you need to know who’s actually visiting and why. The three categories don’t share access consequences — blocking the wrong one closes off citation while leaving training open, which is the most expensive mistake you can make in this layer.

What to read for this step

  1. AI Crawlers
    Wiki
  2. ChatGPT-User
    Wiki · coming soon

After this step, you should be able to answer

  • Name the major AI crawlers and identify each one's User-Agent
  • How can an engine cite you without ever crawling you?
  • Which crawlers train models, and which fetch your page at answer time?