Measurement

Every engine gave a different shortlist. Not one firm appeared on all of them.

By Arnav Mukherjee, founder of TofuBofu · August 5, 2026

On 3 August we ran a batch of discovery queries to find firms for our own research pipeline. Nine questions, put to the six AI engines we track, of which four returned answers that day. They were the sort of thing a buyer types when they have a problem and no shortlist: who are the best personal injury lawyers in Columbus, who are the best legal marketing agencies, and so on across five metros and four professional categories.

I was reading the stored answers for something else entirely when I noticed the four Columbus shortlists had almost nothing in common. Not different orderings of the same firms. Different firms. I checked Tampa, and found the same thing: ChatGPT named seven firms, Claude named six, Bing Copilot named fourteen, Google AI Mode named eight, and exactly three names appeared on as many as three of those four lists.

That is a measurement problem before it is anything else. If you run a firm in Tampa and you check one engine to see whether AI recommends you, what exactly have you learned? So I counted properly across all nine queries, and the answer turned out to depend almost entirely on how specific the question was.

The local result: 136 firms, and not one consensus pick

Five metros, one buying question each, four engines answering. That produced 136 firm-and-market entries, covering 134 distinct firms, since one national brand turned up in three separate markets. Of those 136 entries, 116 were named by exactly one engine. Fifteen were named by two. Five were named by three. Not a single firm, in any of the five metros, was named by all four.

That last number is the one that stopped me. Across five separate markets and twenty separate answers, the four engines never once agreed on a single firm.

A method note, because this is exactly where analyses like this fall apart. Engines write the same firm several ways. Bing Copilot returned "Gunn Law Group P.A." and Google AI Mode returned "Gunn Law Group, P.A.", which is one firm named twice, not two firms named once. We have been bitten by this before: an earlier regional study of ours counted shortened names as separate companies and published firms as invisible when an engine had in fact named them. So the figures above are computed after stripping punctuation and common suffixes such as PA, PLLC, LLC and Injury Attorneys. Before normalisation the raw numbers were 144 firms and 90 percent named by exactly one engine. After normalisation, 136 and 85 percent. The conservative figure is the one in the table.

One more caveat worth stating plainly, because it is the difference between blocked and zero. Six engines were queried; four returned answers. Perplexity and Gemini returned nothing in this collection run for reasons on our side, not because they declined the question. Everything here describes four engines, and none of it should be read as a finding about the other two.

The same engines agree far more on a broader question

The four remaining queries in the batch asked the same engines for the best marketing agencies in four categories: legal marketing, SaaS marketing, B2B SEO, and agencies in Austin. Same engines, same day, same extraction, same normalisation. The behaviour is visibly different.

Those four questions produced 138 distinct agencies, of which 108, or 78 percent, were named by exactly one engine. Still a scattered field, but with a meaningful head: 25 agencies were named by two engines, three by three engines, and two were named by all four. Scorpion was on every engine's legal marketing list. Kalungi was on every engine's SaaS marketing list. In the local queries, the equivalent count was zero.

The mechanism is not mysterious. A category question has a small set of heavily documented answers. Large agencies get written about in directories, roundups, review sites and press, so each engine retrieves overlapping evidence and arrives at overlapping names. Ask about personal injury firms in one metro and the candidate pool is hundreds of small practices, each with a modest and unevenly distributed footprint. Small differences in what each engine indexed produce large differences in what it says.

There is a second effect worth noting. Exactly one firm in the entire local set appeared in more than one metro: a national personal injury brand that turned up in three of the five. Every other name was local to its own market. National advertisers can occupy a slot in many metros at once, but the rest of the field stayed genuinely local, which is a more hopeful reading than the usual one. Most of the slots are still winnable.

Some of that disagreement is not disagreement

Four different shortlists could be four defensible opinions. In a metro with two hundred capable firms, reasonable sources will pick differently, and that would be fine. So I took the single Columbus answer that had first caught my eye and checked all six names by hand.

Four of the six I could find no trace of: no website, no bar listing, no directory entry, nothing in a general web search. One, Schottenstein Zox & Dunn, was a real and well-known Columbus firm, and it stopped existing as an independent firm when it combined with Ice Miller LLP effective 1 January 2012. It was recommended in 2026 as a current option. The sixth, Gallagher Sharp, is a real firm founded in 1912 with a Columbus office, but its practice is insurance and civil defence litigation, which is the opposite side of the docket from the plaintiff-side injury representation the question asked for.

So of six confident, well-formatted recommendations, not one was a currently operating Columbus plaintiff personal injury firm. I want to be careful about the strength of that claim: absence from a web search is not proof that a firm does not exist, and a small practice can have a thin footprint. But four unfindable names, one dissolved fourteen years ago and one on the wrong side of the case is not a near miss.

We have seen this pattern before in a regional managed IT study, where one engine's entire shortlist proved unverifiable. Finding it again in an unrelated profession, with a different question, suggests it is a property of thin evidence rather than a quirk of one industry. The practical consequence for you is simple and slightly bleak: from inside a single answer, a buyer cannot tell an omission from an invention, and neither can you.

What this changes about measuring your own visibility

Four things follow from the numbers above, and they are worth acting on in this order.

None of this makes AI recommendations less consequential. G2's 2026 research found 51 percent of B2B buyers now begin vendor research on an AI chatbot, up from 29 percent, and 69 percent had switched their vendor choice based on what AI told them. A buyer in Columbus gets one of these shortlists, not four, and has no way of knowing how little the other three would have resembled it. The instability is not a reason to dismiss the channel. It is the reason to measure it properly.

The honest limits of this dataset

Nine queries is a small study and I would rather say so than dress it up. It is one question per metro, run once, on one day, by four engines. Engine behaviour shifts with model updates, and a rerun next month would produce different names. Two of our six engines returned nothing in this run, so this is not a six-engine finding.

What I do think survives all of that is the direction and the size of the gap. Two unanimous picks on the category questions against zero on the local ones, from the same engines on the same day with the same processing, is a difference too large to come from sampling noise alone. If you sell into a metro rather than a category, assume your visibility is four different numbers until you have checked.