Getting Cited by AI: Three Technical Details Most Sites Miss
Crawlers see initial HTML, not the rendered page. That single difference decides whether your content gets cited.
Search engines return ten blue links and let the human decide which one to click. Generative engines return an answer. Your site is either cited in it, or it might as well not exist.
The three most commonly missed
1. robots.txt doesn't allow AI crawlers
Most robots.txt files were written years ago with only Googlebot in mind. GPTBot, ClaudeBot, and PerplexityBot all respect robots.txt — if you don't explicitly allow them, they won't come.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
2. Critical content is client-rendered
Crawlers see the initial HTML. If your article body arrives via a useEffect fetch and gets injected into the DOM, they see an empty shell. Server rendering isn't a performance optimization here — it's a precondition for being visible at all.
3. No llms.txt
This is a newer convention: put an llms.txt at your site root listing, in Markdown, the pages you want cited. It isn't required, but it costs almost nothing and states plainly which pages are your substance.
How to check
The most direct method is to visit your own site as a crawler would:
curl -A "GPTBot" https://your-site.com/ | head -50
Look at what comes back. If it's a pile of <script> tags around an empty <div id="root">, you've found your problem.