There is a whole class of traffic to your website that your analytics has never shown you and, in most setups, structurally cannot. Not because it is small — for a documentation site or a technical blog it is frequently a double-digit share of all requests — but because of how the measurement works.
Analytics tools count pageviews with a JavaScript snippet. A crawler does not run JavaScript. The two facts multiply out to a simple conclusion: every crawler visit is invisible to every script-based analytics product, and that has been true since the first tracking pixel.
For twenty years this did not matter much. Crawlers were search engines, search engines sent traffic, and the traffic was the thing you measured. That relationship has now partly broken. An AI assistant can read your page, use it to answer somebody's question, and send you nobody at all.
Who is actually out there
The list changes, but as of now these are the ones that matter, grouped by what they are for — which is the distinction most people miss.
Training crawlers collect text to train models. They are not attached to any user request.
- GPTBot — OpenAI
- ClaudeBot — Anthropic
- Google-Extended — not a crawler at all, but a token in robots.txt that governs whether Google may use already-crawled content for Gemini training
- Applebot-Extended — the same idea for Apple
- CCBot — Common Crawl, which is not an AI company but is the source many models are trained from
- Bytespider — ByteDance
- Meta-ExternalAgent — Meta
Retrieval crawlers fetch a page because a user just asked something and the model wants a source. These correlate with real people.
- OAI-SearchBot and ChatGPT-User — OpenAI, browsing on behalf of a session
- PerplexityBot and Perplexity-User — Perplexity
- Claude-Web / Claude-User — Anthropic, retrieval rather than training
- Amazonbot, Applebot, DuckAssistBot, YouBot
The split matters enormously and almost nobody separates them. Blocking a training crawler costs you nothing today. Blocking a retrieval crawler removes you from answers people are asking for right now.
Where the evidence lives
Since your analytics cannot see any of this, you have three real options.
Server or edge logs
The most reliable source, because the request is recorded whether or not anything executes. Every crawler identifies itself in the User-Agent string, and the major ones publish IP ranges you can verify against — which matters, because User-Agent is trivially spoofed and a meaningful fraction of traffic claiming to be GPTBot is not.
If you are on Cloudflare, this is available in the dashboard under bot analytics without touching your origin. If you run your own server, the raw access log has it and always did.
The drawback: logs tell you a bot fetched a URL. They do not tell you whether it mattered, and they are painful to read over time.
robots.txt request patterns
A weak signal but a free one. Every well-behaved crawler fetches robots.txt before it fetches anything else, so the requests for that one file are a rough census of who is interested in you.
Analytics that sits in the request path
This is the only option that puts crawler activity in the same place as everything else you measure. It requires the measurement to happen at the request rather than in the page, which in practice means the analytics has to run as part of your infrastructure — a Worker in front of the site, a middleware in your app, a log pipeline you own.
That is what edge-mode measurement is, and it is the reason a self-hosted tool can answer a question that no hosted script-based product can.
The number worth building a habit around
Once you can see crawler traffic, the useful thing is not the raw count. It is the ratio between two numbers:
Pages an AI crawler fetched against visits arriving from an AI assistant.
The first is what they took. The second is what they gave back. A page that is read constantly and cites you never is a page doing unpaid work, and until you can see both halves you cannot tell the difference between that and a page nobody has found. That comparison deserves its own treatment — see crawled but never cited.
What to do with the answer
Resist the reflex to block everything. The honest position is that this is a real trade with real numbers on both sides, and whether to block AI crawlers depends on which side of it you are on. What you should not do is decide it blind, which is what almost everyone is currently doing.
Start by measuring. The rest follows from what you find.
Common questions
Why does my analytics show no AI crawler traffic at all?
Because analytics tools count pageviews using a JavaScript snippet in the browser, and crawlers do not execute JavaScript. The request happens, your server answers it, and the snippet never runs — so nothing is recorded. This is not a configuration problem and cannot be fixed with a setting; it is a consequence of measuring from inside the page. Server logs, edge logs and request-path analytics all see it because they record the request itself.
Can I trust the User-Agent a crawler sends?
Not on its own. User-Agent is a plain string that any client can set to anything, and a noticeable share of traffic claiming to be GPTBot or ClaudeBot is neither. OpenAI, Anthropic, Perplexity and Google all publish the IP ranges their crawlers use, and serious measurement verifies the claimed agent against the address it came from. Cloudflare does this verification for you before the request reaches your origin.
Is AI crawler traffic large enough to care about?
It depends entirely on what you publish. A local business site sees very little. Documentation, technical writing, reference material and anything answering a how-to question sees a great deal — it is not unusual for AI crawlers to account for a tenth or more of all requests on such a site. The only way to know your own number is to measure it, which is the point of the article above.