Skip to content
Technical GEO Intermediate
Step 3 of 8

robots.txt and access control

robots.txt is the cheapest, oldest lever you have — and the easiest one to misuse against AI crawlers. The trap is to assume a single rule covers every bot, when each vendor’s compliance posture differs, and the protocol’s allow/disallow precedence has corners that bite at scale.

What to read for this step

  1. robots.txt
    Wiki
  2. Google-Extended
    Wiki · coming soon

After this step, you should be able to answer

  • How do you write a robots.txt rule that allows only specific AI bots?
  • What's the precedence between allow and disallow when they conflict?
  • Which AI bots actually honor robots.txt, and which routinely ignore it?