AI Readable
← All posts

AI Crawlers Come in Three Kinds — Do Not Block Them All

Training, live retrieval, and user-triggered fetches are three different things. Only the second one sends you traffic.

Before you finish filling in the User-agent lines in robots.txt, get one thing straight: those names represent three different behaviors, and they are worth very different things to you.

Three kinds

1. Training

GPTBot, ClaudeBot, Google-Extended, CCBot. These pull content into training corpora and send you no traffic. Whether to allow them is a business call: do you want your arguments inside the model's "common sense," or do you consider that giving away the store.

2. Live retrieval

OAI-SearchBot, PerplexityBot, Claude-SearchBot. When a user asks something, these search and fetch on the spot, then cite you with a link in the answer. This is the category that actually sends traffic. Blocking them is opting out of the generative-search results page.

3. User-triggered fetches

ChatGPT-User, Perplexity-User. A user pasted your link into a conversation and the model went to fetch that page. Someone deliberately asked to see your content; blocking it just hands them "could not access."

A common misconception

Many people assume Disallow prevents their content from being used for training. In practice:

  • robots.txt is voluntary; it has no enforcement
  • It only stops direct fetching from your site — it cannot stop your content being scraped after someone else reposts it
  • Common Crawl (CCBot) snapshots already exist; blocking now only affects what comes next

A configuration worth copying

If your goal is citation traffic — true for most content sites — allowing everything is simplest:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api/

Sitemap: https://your-site.com/sitemap.xml

If you want traffic without donating training data, split them:

# Training: refuse
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Retrieval and citation: allow
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Not every vendor honors this split strictly, but it at least states your intent unambiguously.

How to verify

Do not assume it took effect. Actually request the page as a crawler:

curl -A "OAI-SearchBot" -I https://your-site.com/
curl -s https://your-site.com/robots.txt

Watch two things: whether the status is 200, and whether your CDN treats those user agents as hostile traffic. Cloudflare's "AI Scrapers and Crawlers" block is on by default — plenty of sites have a perfectly good robots.txt and are still blocked one layer above it.

If you would rather not check them one by one, the AI crawler access tool lists twelve crawlers and tells you which are allowed, which are blocked, and which rule in your robots.txt decided it.