Bots and AI crawlers in 2026: how many, and who gets disallowed in robots.txt

"Bots are now the majority of traffic" is true and false at once, and the difference is not the source but what gets counted. Cloudflare Radar gives two answers for the same week (20–26 August 2026):

The HTML breakdown goes further: non-AI bots 52.8%, humans 38.1%, AI bots 5.7%, mixed-purpose 3.4%.

The reason is simple: a person pulls dozens of images and scripts per page, while a crawler usually takes the HTML alone. Counted in requests, humans win; counted in pages, they lose.

The denominator is mandatory. Any bot-share figure that does not say whether it counts all requests or only pages is misleading — regardless of who published it.

Cloudflare found 4,365 robots.txt files among the ten thousand largest domains (data as of 24 August 2026) and published the rank order of the most-disallowed AI agents: GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, meta-externalagent, Amazonbot, Applebot-Extended. Their per-agent counts are encoded in a chart in a way that cannot be read unambiguously, so we give the order only.

We measured the same thing on our own sample — 600 hosts, 150 per zone, on 27 August 2026, parsing groups per RFC 9309 (consecutive User-agent lines share one rule set):

A caveat without which the table must not be read: each zone has only 4–10 blocking sites. At those counts the zones cannot be told apart — the spread from 8.2% to 12.5% sits inside chance. Only one thing here is solid: roughly one site in ten that has a robots.txt at all closes itself to AI crawlers, and that holds equally across all four zones.

A side observation the numbers support better: robots.txt is more common on .ru sites than on .com — 97 against 85 out of 150.

We counted which agents are mentioned at all in the collected files — not necessarily disallowed, simply named. There were 230 distinct agents. The top of the list (sites out of 600):

Seven of the ten places belong to AI crawlers, and GPTBot outranks both Googlebot and Yandex. That is indirect but telling: site owners now write rules more often for the machines that train models than for the machines that bring visitors.

On Yandex specifically: its agents appear across all four zones, but in our data no site closes access to it outright — the mentions come alongside indexing settings, not prohibitions.

A Cloudflare press release dated 13 July 2026 puts automated traffic at roughly 57%, and a 1 July blog post says "more than 50% of traffic on the Internet is now non-human". Neither states a measurement period, a population or a methodology, and neither reconciles with Cloudflare Radar’s own data for the same months (35.5% across all requests).

There is a counterweight too. Akamai — a vendor with every commercial incentive to say the opposite — puts AI bots at nearly 1% of all bot traffic on its network. The widely quoted "300% growth in AI bots" is measured from that base, but retellings never carry the base itself.

The general rule for this topic: every available figure measures a particular vendor’s customer base, not the internet. Cloudflare sees sites behind Cloudflare; Imperva sees those who bought bot protection, i.e. those who already had a problem; Fastly sees its own applications. No neutral measurement of the whole internet exists, and that deserves saying out loud.

The easiest way to see what your own site gets is your logs. To check the file itself is well-formed, use the robots.txt checker.