AI crawlers read your site for AI engines. If you block the crawler an engine uses to answer questions, it cannot cite you. Allow the answer crawlers (OAI-SearchBot, PerplexityBot, Bingbot) so you stay citable, and block the training-only ones (GPTBot, Google-Extended, CCBot) if you would rather your content did not train models.
What are AI crawlers?
AI crawlers are the bots that fetch your pages on behalf of AI engines. They come in three flavours, and the difference is what decides whether blocking one helps or hurts you:
- Training crawlers collect content to train the underlying models.
- Search or answer crawlers index your pages so the engine can find and cite you when it answers a question.
- User-triggered fetchers grab a specific page in real time because a user asked about it.
Why does crawler access matter for AEO?
Because access is the gate before everything else. All the answer-first structure and authority in the world means nothing to an engine that is not allowed to read the page. If your robots.txt blocks the crawler an engine uses to build answers, that engine simply cannot use you as a source, no matter how good the content is. Getting access right is the first, and most overlooked, step in answer engine optimisation.
Which crawlers should you know?
The main AI user-agents, and crucially whether each is for training or for answering:
| Crawler | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training |
| OAI-SearchBot | OpenAI | ChatGPT search / citations |
| ChatGPT-User | OpenAI | User-triggered fetch |
| PerplexityBot | Perplexity | Search / citations |
| Perplexity-User | Perplexity | User-triggered fetch |
| ClaudeBot | Anthropic | Training / retrieval |
| Google-Extended | AI training and grounding | |
| Bingbot | Microsoft | Bing index (powers Copilot) |
| CCBot | Common Crawl | Open dataset (training) |
The operators document these directly: OpenAI's crawlers, Perplexity's crawlers, and Google's note that appearing in its AI features needs no special files, just a crawlable, indexed page.
How do you allow or block them in robots.txt?
In robots.txt, at the root of your domain, with a rule per user-agent. To allow everything, you simply do not disallow. To block a specific crawler, add a block for its user-agent. For example, to keep your content out of model training while staying citable in search answers:
# Block training crawlers User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / # Allow the answer / search crawlers (default is allowed) User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: /
One caveat: the user-triggered fetchers (ChatGPT-User, Perplexity-User) treat requests as user actions and generally ignore robots.txt, because a person asked for that exact page in the moment. You control the scheduled crawlers; you cannot fully block a real-time user fetch this way.
Should you block training but allow search?
For most sites that want AI visibility, yes. This is the key decision, and it is not all-or-nothing. The training crawlers and the answer crawlers are separate user-agents, so you can:
- Block training (GPTBot, Google-Extended, CCBot) if you would rather your content did not feed model training.
- Allow the answer crawlers (OAI-SearchBot, PerplexityBot, Bingbot) so those engines can still find and cite you.
The mistake is blocking with a broad brush, disallowing everything AI-related, and quietly removing yourself from the answers you wanted to appear in. If being cited matters, keep the answer crawlers in.
What else can stop a crawler seeing you?
Access is not only robots.txt. Two other things quietly hide your content:
- JavaScript-only content. If your words only appear after a browser runs JavaScript, many crawlers never see them. The text must be in the raw HTML, view source and check.
- Server or CDN blocks. A firewall or CDN rule that blocks by user-agent or region can stop an AI crawler even when robots.txt allows it. This is a common accidental block.
How does Limecube help?
Crawler-access problems are silent, you only notice when you are not being cited and cannot work out why. Limecube's Site Audit checks the access basics for you: whether the answer-engine crawlers are allowed, whether your key content is in the raw HTML rather than injected after load, and where a page is quietly unreadable, before it costs you a citation. The 7-day trial covers it.