Measurement

Software and IT services sites lost up to 40% of their human visitors. Here's what a B2B firm should change.

By Arnav Mukherjee, founder of TofuBofu · October 8, 2026

In September a marketer on r/SEO posted that they'd added a middleware log to separate GPTBot, ClaudeBot and PerplexityBot hits from normal traffic. Then they asked the room the question that matters: "what did you actually learn from the logs?" The thread drew 26 comments.

Two weeks later, Cloudflare answered part of it from the other side of the wire. Its post "The Internet has a second audience" says that this year, for the first time, more than half of Internet traffic wasn't human. It names four heavily crawled categories losing human visitors, and two of them are the market we serve.

What did Cloudflare measure, and what didn't it?

Read the numbers with their frame attached. Every figure below is Cloudflare measuring traffic on its own network, published 30 September 2026:

• Share. More than half of Internet traffic wasn't human this year, the first time that's happened.
• Growth. Daily requests from AI agents grew by more than 1,700% over the past year.
• Purpose. AI training was 22% of the crawler requests Cloudflare saw in spring 2025, and 52% by June 2026.
• Who's losing people. "Some of the most heavily crawled categories, like Retail, Computer Software, IT & Services, and Financial Services, have seen human traffic decline as much as 40% in less than one year."

Three limits, stated so you don't overread it:

• "As much as 40%" is a ceiling. It's the top of the range for the most heavily crawled categories. Your site may have lost less, or nothing.
• The method isn't in the post. Cloudflare doesn't say how it sorted sites into categories or measured the decline.
• It's one network. Cloudflare carries a large slice of the web. It's still a sample, and it isn't your analytics.

Even with those limits, the direction matters for a B2B firm. Cloudflare puts software and IT services among the most heavily crawled categories, and among those losing human visitors.

Does a traffic drop mean demand dropped?

You can't tell from sessions, and that's the change worth absorbing. Cloudflare describes an agent as software fetching pages on a person's behalf, "often because the human asked a chatbot." Picture how that plays out for a services firm:

1. A founder asks ChatGPT for three managed IT providers near them.
2. The engine fetches a handful of service pages, yours among them, and writes a shortlist.
3. The founder reads the shortlist and emails two names.
4. Unless the founder clicks through, your analytics record no visit from them at all.

The research happened. Your page was read. The visit didn't happen. G2's 2026 report found 51% of B2B software buyers now begin their software research with an AI chatbot more often than with Google, so in software this path is already the common one. Our read is that services buyers follow within a year or two.

So a falling session count has two readings. Fewer people want what you sell, or more of the reading now happens on their behalf. Both produce the same line on a GA4 chart. Only one of them is a pipeline problem.

Should a B2B firm block AI bots in Cloudflare?

On 1 July 2026 Cloudflare replaced its single "block AI bots" switch with three categories, on every plan including Free. Its changelog defines them:

• Search: "Crawlers that index your content so they can answer questions about it later, where you should expect referral traffic or other equitable compensation in return."
• Agent: "Automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents."
• Training: "Crawlers that take your content to train or fine-tune a model."

For each one you can block on all pages, block only on pages that display ads, or not block. Domains that join Cloudflare on or after 15 September get new defaults: Training and Agent blocked on pages that display ads, Search allowed. Existing domains keep their current settings, so check yours.

Cloudflare's post makes the case for ad-supported sites: "an ad only pays when a person sees it." A B2B services firm earns nothing from a pageview. It earns from being on the shortlist. Our settings for that kind of firm, stated as our position:

• Search: allow. This is how an engine knows your service page exists when the question comes.
• Agent: don't block. By Cloudflare's own definition, a person is on the other end in real time. For you, that person is usually a prospect.
• Training: decide deliberately. There's a fair case for keeping proprietary research out of training sets. But Cloudflare's changelog says multi-purpose crawlers that combine Search and Training are affected when Training is blocked, so find out which crawlers you'd lose before you flip it.

Then open the setting and read it. If you run ads or retargeting on any page, ask whether Cloudflare counts that page as one that displays ads, because its changelog doesn't say how it decides. Robots.txt rules still matter for crawlers that name themselves; our robots.txt guide covers which ones do.

What should you measure instead of sessions?

Back to the r/SEO question: what do the crawler logs teach you? They tell you who read the page. They can't tell you what the engine said afterwards, and that's the part a buyer acts on. Keep the logs. Add three measures they can't give you:

• Answers that name you. Put your buyers' real questions to the engines they use and record whether each answer names you, a rival, or nobody. Hold the questions fixed month to month so a change is a change in the engines and not in your test.
• AI referrals, read as a floor. The clicks that do come from an AI answer carry a referrer or a utm_source. Count them, and remember an agent reading on someone's behalf rarely produces one. Here's how to set that up in GA4.
• How leads say they found you. Add "an AI assistant" to the source question on your contact form. It's crude, and it's the one place the agent path shows up in your own data.

SEO is still the floor: an engine that retrieves has to find your page first. Being named in the answer is a separate layer, and you can win it when your Google rank doesn't move. A session report measures neither layer well any more.

Last, make the page easy to read for the agent that does arrive. Say who you serve in a sentence, put prices and terms in text, and keep headings in order. Our guide to agents that don't identify themselves has the full list. When the reader is software working for a buyer, a page it can parse is the pitch.

Frequently asked questions

Sources and further reading

• Cloudflare: The Internet has a second audience, 30 September 2026, read at source 8 October 2026: the non-human share, AI agent growth, the training share of crawling, and the categories losing human traffic
• Cloudflare changelog: AI traffic categories, 1 July 2026, read at source 8 October 2026: the Search, Agent and Training definitions and the 15 September defaults
• G2: The Answer Economy, 2026 AI Search Insight Report, n=1,076 B2B software buyers, surveyed March 2026: 51% begin software research with an AI chatbot more often than with Google
• r/SEO: Started logging AI crawler traffic separately from Google. Does it work?, September 2026: the question this post opens with