AI crawlers are not blocked by robots.txt

Why this matters

This is the precondition for everything else. Block the AI crawlers and your content never reaches model training data or live retrieval, which makes every other optimisation moot. These blocks often come from a CDN default template that the site owner never chose.

How to fix it

Which advice applies depends on what the audit found. The label is the condition that triggers it.

• blocked

  Explicitly allow the paths you want indexed for these user-agents. If the block comes from a CDN "AI crawler protection" default (Cloudflare, Akamai and friends), it has to be turned off in the CDN console.

Try it on your site

• AI crawler access

Other AI crawler access rules

• The page returns 200
• robots.txt is reachable
• sitemap.xml exists and is declared
• llms.txt site guide
• Response speed