Training vs. retrieval AI crawlers: which ones you can safely block

A site owner reads that AI companies are training on the open web, decides they would rather not participate, and adds a few lines to robots.txt. That rule can also block search crawling if it catches the agents assistants use to discover sources.

That outcome is not a policy decision going wrong. It is one crawler being mistaken for another. This guide is about telling them apart.

Three jobs, not one

"AI crawler" describes at least three different things, and the consequence of blocking one has nothing to do with the consequence of blocking another.

Training crawlers

These collect content that may be used to train models. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, whose public dataset many models train on), Bytespider, meta-externalagent and Amazonbot are in this group.

Training rules govern future collection by crawlers that honour them; they are not a deletion request for content already collected. Whether you want to is a genuine editorial and commercial judgement, and there is an honest case on both sides. It is also the decision most people think they are making when they write an AI crawler rule.

Retrieval crawlers

These fetch and index pages so an assistant can search them when answering a live question. OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot (Perplexity) are in this group.

Blocking search crawlers can restrict discovery and search citations. OpenAI says sites that opt out of OAI-SearchBot are excluded from ChatGPT search answers but may still appear as navigational links. That is narrower than disappearing from every AI answer. This is almost never what someone intends, and it is the failure this guide exists to prevent.

User-requested fetchers

The category most guides omit. ChatGPT-User, Claude-User and Perplexity-User fetch a specific page because a human pasted a link or asked the assistant to go read it. There is no crawl and no index — it is one fetch, on demand, for someone who explicitly asked.

Do not assume robots.txt blocks user-requested fetches. OpenAI notes that its robots.txt rules may not apply to user-initiated requests; Perplexity says its user fetcher generally ignores them. Access controls at your server are a separate mechanism.

The pairs that get confused

Read the OpenAI rows together. GPTBot and OAI-SearchBot differ by one word and produce opposite outcomes. The same is true of ClaudeBot and Claude-SearchBot.

Google-Extended is not a crawler

Worth its own section, because it is the most expensive misunderstanding here.

Google-Extended does not fetch anything. It is a permission token that controls whether content Googlebot has already crawled may be used for Gemini training and for grounding AI answers, per Google's crawler documentation. Googlebot is a separate agent and handles Search.

Disallowing Google-Extended does not affect your Google Search indexing or ranking. If you have avoided it out of fear of a ranking penalty, that fear is misplaced. Applebot-Extended works the same way for Apple: plain Applebot continues to power Siri and Spotlight.

The robots.txt rule that surprises people

Even with the right agents identified, robots.txt has one precedence rule that inverts results:

A group naming a specific crawler is used instead of the User-agent: * group, not in addition to it.

So this file does not block GPTBot:

User-agent: *
Disallow: /

User-agent: GPTBot
Allow: /

GPTBot reads its own group, finds Allow: /, and crawls the site. The blanket rule never applies to it. This works in both directions, and it is why "I added a global disallow, surely that covers everything" is often wrong.

Two more rules worth knowing:

• The longest matching path wins, regardless of line order, with Allow taking ties. Disallow: / plus Allow: /blog/ blocks the site but permits the blog.
• Consecutive User-agent: lines share the rules beneath them, so several agents listed back to back are governed by one block.

What to actually write

If the intent is "don't train on my work, but do cite me" — which is what most people mean — disallow the training agents by name and say nothing at all about the retrieval ones:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

The silence is deliberate. Adding Allow: / for OAI-SearchBot achieves nothing on its own — absence of a rule is already permission — and an unnecessary Allow can narrow a wildcard rule defined elsewhere in the file. Emit denials only.

If instead you want out of AI entirely, including the answers, block both groups and accept the trade knowingly. That is a legitimate position; it should just be the one you chose.

Checking what you actually have

Robots.txt files accumulate. Rules get added by plugins, security tools and previous developers, and the interaction between a wildcard group and a named one is not obvious by reading. Our AI Crawler & robots.txt Checker fetches a site's file, applies the same precedence rules real crawlers use, and reports each agent with the line that decided it — grouped by what the block actually costs.

Being reachable is the floor, not the ceiling

Getting this right does not get you cited. It removes one reason not to be. A page that every crawler can reach still needs a passage worth lifting — which is a different problem, covered in how answer engines choose which sources to quote.

Reachability is worth fixing first only because it is binary: no amount of good writing survives a Disallow.