Industry surveys have put automated traffic at roughly half of all requests on the web for several years running. On a small site with a handful of human visitors a day, the proportion is often much higher — a new domain can spend its first month being read almost exclusively by machines.
If your analytics counts them, every conclusion you draw is contaminated. Here is what is actually arriving and what to do about it.
What is out there
Search crawlers. Googlebot, Bingbot, and dozens of smaller ones. Honest: they identify themselves in the user-agent and respect robots.txt. Every analytics tool filters these.
AI training and retrieval crawlers. GPTBot, ClaudeBot, PerplexityBot, CCBot and a growing list. Mostly honest about identifying themselves, and a rapidly growing share of total traffic since 2023.
Uptime and security monitors. Pingdom, UptimeRobot, and every scanner probing for a vulnerable WordPress installation. Frequent, regular, and often hitting the same path every sixty seconds.
SEO and marketing crawlers. AhrefsBot, SemrushBot, MJ12bot. Aggressive, and they crawl deeply.
Headless browsers and scrapers. This is the category that matters. A scraper running headless Chrome executes JavaScript, has a plausible user-agent, loads your analytics script and gets counted as a human being. Some are price monitors, some are content thieves, some are people building datasets.
Link preview fetchers. When a URL is pasted into Slack, WhatsApp, iMessage or a social platform, a fetcher retrieves the page to build a preview card. One shared link can produce dozens of these.
Why user-agent filtering is not enough
Most analytics tools maintain a list of known bot user-agent strings and drop matching requests. This works exactly as well as the bot's honesty allows.
The user-agent header is a string the client chooses. A scraper that wants to look like Chrome on macOS says it is Chrome on macOS, and no list will ever contain it. Meanwhile, the well-behaved crawlers that do identify themselves are the ones causing you the least trouble.
So user-agent filtering removes the traffic you did not mind and keeps the traffic that distorts your numbers.
What works better
Network-level signals. The infrastructure in front of your site sees things JavaScript cannot: the TLS handshake fingerprint, HTTP/2 frame ordering, the reputation of the originating network, and whether this address has been seen doing the same thing to thousands of other sites in the last hour. Cloudflare, Fastly and similar networks all expose a bot score derived from these. It is far harder to forge a TLS fingerprint than a user-agent string, and a bot on a datacentre ASN cannot pretend to be on a home broadband connection.
Behavioural signals. A visitor that requests forty pages in nine seconds, never moves a pointer, never scrolls, and has zero engaged time on every page is not reading anything. Requiring a small amount of visible time before counting a pageview removes a great deal of automation at almost no cost to real readers.
Prefetch and preview awareness. Browsers speculatively load pages the user might visit next, and Sec-Purpose: prefetch says so. Counting a prefetch as a pageview inflates the pages that happen to be linked from popular pages.
Your own traffic. Your staging domain, your office network, your own repeated visits to the page you are working on. Every one of these should be excluded, and almost nobody sets it up on day one — it is the first thing to check when two tools disagree about your traffic.
What to expect when you turn it on
Numbers will fall. On a low-traffic site they may fall a lot — it is entirely normal for a new blog to discover that most of its "audience" was automated.
This is a correction, not a loss. The traffic was never there. What you gain is the ability to tell whether an article actually found readers, which is impossible when the signal is buried under crawlers.
The clearest way to see it: compare your traffic graph before and after filtering. Bot traffic is flat and evenly distributed across the day, because machines have no timezone. Human traffic has a shape — a morning rise, a lunchtime dip, an evening peak, a weekend that looks different from a Tuesday. If your graph is a flat line, you are looking at machines. If a shape appears after filtering, the filtering worked.
What not to do
Do not block them. Filtering analytics and blocking requests are different decisions with different consequences. Blocking search crawlers removes you from search results. Blocking AI crawlers is a legitimate editorial choice, but it belongs in robots.txt and your WAF, not in your analytics configuration.
Do not filter by IP allowlist alone. Residential proxy networks give scrapers home-broadband addresses. Address reputation is one signal among several, not a verdict.
Do not compare filtered numbers to unfiltered history without noting the change. If you turn on strict filtering in March, your February comparison is meaningless. Annotate the date, and remember that comparisons only work when the method is constant.
A reasonable configuration
- Filter at the network edge first, using the platform's bot signals — this catches the dishonest ones.
- Keep a user-agent list as a second layer, for the honest ones that never reach the edge filter.
- Require a small amount of visible time before recording engagement.
- Ignore prefetch and preview requests.
- Exclude your own networks and your staging hostnames.
- Record the date you changed any of this, and annotate the chart.
The goal is not zero bots. It is a number stable enough that a change in it means something happened.
Common questions
How much of my website traffic is bots?
Industry measurements have put automated traffic at roughly half of all web requests for several years. The share is usually higher on low-traffic sites, where a handful of crawlers, uptime monitors and scrapers can outnumber human visitors many times over.
Why doesn't user-agent filtering catch all bots?
The user-agent is a string the client chooses, so a scraper that wants to look like Chrome simply says it is Chrome. User-agent lists only catch bots that identify themselves honestly, which are generally the well-behaved ones. Network-level signals such as TLS fingerprints and network reputation are far harder to forge.
Should I block bots as well as filter them from analytics?
They are separate decisions. Filtering keeps automated traffic out of your statistics; blocking stops the request entirely. Blocking search crawlers removes you from search results, so blocking belongs in robots.txt and your firewall rules as a deliberate editorial choice, not as a side effect of an analytics setting.
How can I tell whether my traffic is human?
Look at the shape of the daily curve. Human traffic follows a timezone — a morning rise, a midday dip, an evening peak, and weekends that differ from weekdays. Automated traffic is flat and evenly spread across all 24 hours, because machines have no schedule.