Most robots.txt advice for AI crawlers is a list of agent names to paste in. The list is the easy part. What causes real damage is two rules about how the file works that are widely misunderstood.
Rule one: a named group replaces the wildcard, it does not extend it
This is the single most common robots.txt mistake, and it is expensive.
A crawler reads the file, finds the most specific group matching its own name, and obeys that group only. Everything under User-agent: * is then irrelevant to it.
So this file does not do what its author thinks:
User-agent: *
Disallow: /admin
Disallow: /cart
User-agent: PerplexityBot
Allow: /The author meant "everyone stays out of admin and cart, and Perplexity is additionally welcome". What they wrote is "everyone stays out of admin and cart; Perplexity may crawl everything, including admin and cart".
Every restriction you want a named agent to obey must be repeated inside its own group.
Rule two: robots.txt is a request, not a control
It is a published document asking well-behaved clients to stay away. The major operators — OpenAI, Anthropic, Google, Apple, Perplexity, Common Crawl — honour it, generally within a day or two.
It does nothing about a scraper using a generic User-Agent, a residential proxy pool, or a browser string. If your requirement is enforcement rather than request, that belongs at the network or edge layer, where you can act on verified identity and rate. Robots.txt is where you state intent.
A working file
This one blocks training, allows retrieval, and keeps the private paths private in every group.
# Everyone
User-agent: *
Disallow: /admin
Disallow: /account
Disallow: /api/
Allow: /
# ---- Training: no ----
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
# Not a crawler - governs Gemini training and grounding
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# ---- Retrieval: yes, minus the private paths ----
User-agent: OAI-SearchBot
Disallow: /admin
Disallow: /account
Disallow: /api/
Allow: /
User-agent: ChatGPT-User
Disallow: /admin
Disallow: /account
Disallow: /api/
Allow: /
User-agent: PerplexityBot
Disallow: /admin
Disallow: /account
Disallow: /api/
Allow: /
User-agent: Perplexity-User
Disallow: /admin
Disallow: /account
Disallow: /api/
Allow: /
Sitemap: https://example.com/sitemap.xmlNote the repetition in the retrieval groups. It looks redundant and it is load-bearing — see rule one.
Whether this particular split is right for you is a business question with measurable inputs, and the agents behind each name are worth understanding individually.
Things worth knowing
Googlebot stays allowed. Blocking it removes you from search. Google-Extended is the AI token and is unrelated to indexing.
You cannot exclude yourself from AI Overviews while remaining in search. No token exists. Why that is, and what to do instead.
Path-level rules are underused. Documentation open, paid course closed, is perfectly expressible and hardly anyone does it.
A blanket AI block leaks through CCBot if you forget it. Common Crawl is not an AI company but is a major training corpus.
Changes take days, not minutes. Crawlers cache robots.txt. Do not conclude a rule failed because you saw a fetch an hour later.
What about llms.txt?
A proposed convention for a Markdown file describing your site to language models. Worth knowing about, currently honoured by essentially nobody, and not a substitute for anything above. The longer version.
Verify
After changing the file, confirm two things: that the crawlers you meant to keep actually still fetch, and that the ones you blocked stop. Both are visible in crawler measurement, and a rule that silently blocked more than intended is the failure mode you want to catch in week one rather than in month six.
Common questions
Why do I need to repeat Disallow rules in every user-agent group?
Because a crawler obeys only the most specific group matching its own name, and ignores the wildcard group entirely once it finds one. If you disallow /admin under User-agent: * and then write a named group for PerplexityBot with only Allow: /, you have granted that crawler access to /admin. Every restriction has to appear inside each group it should apply to.
Does robots.txt actually stop AI companies from scraping?
It stops the ones that identify themselves and honour it, which includes OpenAI, Anthropic, Google, Apple, Perplexity and Common Crawl. It has no effect on scrapers using generic user agents or residential proxies, because it is a published request rather than an access control. Enforcement requires acting at the network or edge layer on verified identity.
How long does a robots.txt change take to have an effect?
Usually one to three days. Crawlers cache the file rather than re-fetching it before every request, so seeing a crawl shortly after you publish a change does not mean the rule failed. Give it several days before concluding anything, and confirm with crawler-level measurement rather than by watching for a single request.