Most advice about AI crawlers treats them as one thing. It is the single most expensive mistake in this area, because the crawlers do two completely different jobs and the case for blocking is strong for one and self-defeating for the other.
Training versus retrieval
A training crawler collects text that may be used to train a future model. There is no person waiting. Nothing you gain from the visit arrives today, or next month, and possibly ever.
A retrieval crawler fetches your page because somebody has just asked an assistant a question and the model wants a source to answer from. There is a person on the other end of it, reading right now, and your URL may appear as a citation they can click.
Blocking the first costs you nothing immediate. Blocking the second removes you from answers being given to interested people at the moment they are interested. They are not the same decision, and a single robots.txt rule that catches both treats them as if they were.
The agents, one at a time
OpenAI
OpenAI runs three, and they are properly separated — which is unusually considerate and worth taking advantage of.
- GPTBot — training. Blocking this removes your content from future model training.
- OAI-SearchBot — building the index behind ChatGPT search. Blocking this removes you from being found.
- ChatGPT-User — a live fetch triggered by a user's request in a conversation. Blocking this means that when somebody explicitly asks ChatGPT about your page, it cannot read it.
Blocking all three because you dislike training is a common and costly move. The three tokens exist precisely so that you do not have to.
Anthropic
- ClaudeBot — the general crawler, historically used for training.
- Claude-User — fetches on behalf of a user in conversation.
- Claude-SearchBot — indexing for search results inside Claude.
Perplexity
- PerplexityBot — indexing for the answer engine.
- Perplexity-User — a live fetch for a specific user question.
Perplexity is almost entirely a retrieval product. Blocking it is close to a pure loss of referral traffic, and Perplexity is one of the more generous surfaces for actually sending clicks through.
Google is the awkward one, because the crawler and the AI use are decoupled.
Googlebot is your search index. You will not block it. Google-Extended is not a crawler — it is a robots.txt token that controls whether content Googlebot has already fetched may be used for Gemini training and grounding. You can set Google-Extended to disallow and remain fully indexed in search.
What you cannot do is opt out of AI Overviews while staying in search. There is no token for that. Overviews are built from the search index, and leaving the index means leaving search.
The rest
Applebot-Extended mirrors Google's arrangement for Apple Intelligence. CCBot is Common Crawl — not an AI company, but the corpus a great many models are trained from, so blocking AI crawlers while allowing CCBot is a gap worth knowing about. Bytespider (ByteDance) and Meta-ExternalAgent (Meta) are training crawlers with poor reputations for volume.
A robots.txt that reflects the distinction
Block training, allow retrieval:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /Two things to know about this file. Each named group replaces the wildcard group rather than adding to it, so any rule you have under User-agent: * does not apply to a named agent unless you repeat it. And robots.txt is a request, not a control — it is honoured by the operators above and ignored entirely by scrapers that do not identify themselves.
For a fuller treatment of the file itself, see robots.txt for AI crawlers.
Measure before you edit
Every rule above is a guess until you know your own numbers. Which of these agents actually fetch your pages, how often, and which ones send anybody back — those are measurable, and they will not match your assumptions. Start by seeing who is there.
Common questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. Being found and cited inside ChatGPT depends on OAI-SearchBot and ChatGPT-User, which are separate tokens. Blocking GPTBot alone keeps your content out of future training while leaving you visible in ChatGPT's search and browsing. Blocking all three is what removes you.
Can I opt out of Google's AI Overviews without leaving search?
No, and this is the honest answer nobody likes. Google-Extended controls Gemini training and grounding, but AI Overviews are generated from the ordinary search index. There is no robots.txt token that removes you from Overviews while keeping you in the results, because they are built from the same crawl. The only opt-out is leaving the index.
Do AI crawlers obey robots.txt?
The named ones from OpenAI, Anthropic, Google, Apple, Perplexity and Common Crawl do, and are generally quick to comply. Robots.txt is a published request rather than an access control, though, so it does nothing about scrapers that use a generic User-Agent or disguise themselves as a browser. If you need enforcement rather than a request, that has to happen at the network or edge layer.