Measurement
An engineer says every AI visibility tool is lying to you. All six of his charges land.
By Arnav Mukherjee, founder of TofuBofu · August 20, 2026
On 12 August I recomputed a number we had already published on eight of our own pages and found it inflated by roughly 2.6 times. The version we shipped said our strongest engine named the scanned brand in about 19 percent of the questions we put to it, with the rest of the field around 10 percent. The corrected figure for that same engine is 7.4 percent.
Nothing had changed at the engines. Our denominator had included brand-name probes, the ones that ask whether a named company is any good. Every engine answers those with the company most of the time, because the answer is sitting inside the question. Put enough of them in the pot and any brand looks healthy.
Six weeks before I found that, a software engineer had already posted the mechanism on the AEO subreddit, under a title that did not invite a friendly reply: "Im an experienced software engineer and every AI visibility tool is lying to you!" He listed six failure modes in six lines, refused to elaborate, and closed by saying tools should not claim "precise, stable numbers without showing the underlying distribution, methodology, variance, and raw evidence." One of those six lines was the bug I would find in our own code six weeks later. He is right about the other five too. Here is the scoreboard, and then the mechanisms, because a concession without a mechanism is just good manners.
The six charges, and where each one lands
Four of the six have been true of our product at some point this year. I can give you the dates on all four, which is the only reason this article is worth more than a press release.
| The charge | True? | Have we done it |
|---|---|---|
| Front-end scraping is unreliable | Yes | No, we do not scrape. Our instrument has different biases, not fewer |
| LLM non-determinism | Yes | Partly. We sample and vote, and we compute a confidence we show to nobody |
| API versus real app gap | Yes | Yes, structurally, and we published the measurement of it |
| Limited prompt sets and scoring tricks | Yes | Yes. Inflated by 2.6x on eight pages until 12 August 2026 |
| Geography is usually ignored | Yes | Yes. Multi-country brands collapsed to one country until 17 August 2026 |
| Model drift | Yes, and understated | Yes. Our own measurement moved on four dates in nine days |
Read as sentences: scraping bias is real and is not our failure mode. Non-determinism is real and we only half answer it. The API gap is real and permanent. Narrow denominators are real and we shipped one. Ignored geography is real and we shipped that too. Drift is real and larger than the charge says.
There is no clean instrument. There are only declared ones.
His first and third charges are one argument. A tool either drives the consumer chat interface with automation, or it calls the API. He is blunt about the first: a scraped answer carries one account, one memory state, one subscription tier, one browser session and one geography, and at any real volume the work runs from cloud machines and proxy routes whose patterns can bleed into the measurement. A buyer typing a specific, messy question into their own logged-in account and a clean browser typing a tidy category question from a datacentre are not the same instrument.
That is correct, and the tempting move for us is to say we call APIs and are therefore fine. We are not fine. A commenter in the same thread put the counterweight better than most vendors would dare: the API plus a search tool is a third instrument with its own bias, not a control group. Pin a model snapshot and the consumer product is very likely serving something else that week. Memory and custom instructions shape what a returning buyer sees and ride along with no bare API call.
So we chose the API and we wrote down what that costs. We do not see personalisation, we do not see interface-only furniture, and we cannot tell you what one specific returning customer sees. What we can do is ask the same question the same way every month and read the movement, which the scraped instrument cannot promise, because its state drifts on its own. We published the comparison rather than the claim, including the part that flatters nobody: the same API call, repeated ten times, returns wildly different company sets. Name your instrument. Publish its blind spots. Anyone telling you they found the neutral one has not looked.
Charge four is the one we committed, and it is the easiest lie in the category
"Limited prompt sets plus scoring tricks" is one line in his post and it is the whole business model of a thin wrapper. You cannot audit a number whose denominator is private, and the denominator is where a visibility score is made or faked. It takes no dishonesty at all to inflate one. It takes an unexamined default.
Ours was brand-sentiment probes. A scan asks a few questions of the form "is this company any good", because sentiment is a real thing to measure and those questions measure it well. They are useless as visibility, because the engine is being handed the name. Left in the pot they lifted every published engine rate by about 2.6 times, and the inflated numbers reached production on all eight of our engine guides and in eleven places on the solutions pages before anyone recomputed them.
The corrected figures, counting buying questions only: our strongest engine names the brand being scanned on 7.4 percent of buying questions, over 46 completed reports covering 2,421 answered cells, on a sample selected on the problem itself, because a firm books a scan when it already suspects the engines are ignoring it. Every other engine we query sits under that, one of them at a fraction of a percent. Those are worse numbers and they are ours, and we now publish the correction on the pages that carry them rather than quietly swapping the digits.
One thing follows from that for any report you are handed, including ours. Ask which question types are in the denominator. If the answer includes anything containing your own company name, you are not looking at whether AI recommends you. You are looking at whether AI can find you, and almost everyone passes that. We wrote the longer version of this argument in our answer to the snake-oil accusation, with the full gradient by question type.
Read the evidence, not the grade
A free scan puts your actual buying questions to all six AI engines, reports each engine on its own, and prints every answer in full beneath the verdict so you can audit it yourself.
Run your free scanGeography is not a filter. It decides which questions exist.
"Geography is usually ignored" is his fifth charge and the one most people underrate, because they picture it as a display setting. Start with the engines' own plumbing. OpenAI's web search documentation states that "to refine search results based on geography, you can specify an approximate user location using country, city, region, and/or timezone." Location is an input the vendor exposes deliberately. A tool that never sets it is not neutral about geography, it has silently chosen one.
We do not set it, and that is a choice I will defend rather than hide. We put the geography in the question instead, because that is what a buyer does. Nobody in Vancouver types "best managed IT services" and relies on the engine to guess. They type the city. So each brand is classified into one of four market scopes, global, national, local, or national with a local angle, and a different family of buying questions is generated for each. That decision sits upstream of everything: get the scope wrong and you have not mislabelled the report, you have spent the entire question budget interrogating the wrong market.
Which is what we did. Three faults stacked. The classifier's schema had no way to return "national" at all while the instructions above it preferred that answer. A regular expression then forced "national" for a list of service industries our own documentation places in the national-plus-local bucket. And the prompt builder took the first four offices from the brand's location list and told the model to infer the single country from what survived. A customer selling into seven markets lost three of them before the prompt existed, and then had the remaining four flattened into one country.
That shipped. It reached a customer's screen as a scan header reading "across one country" followed by a list of seven. The repair, on 17 August 2026, was to pull the whole decision into a small module with its own tests, where "national" now requires exactly one country and a firm whose offices span more than one is reclassified rather than truncated. The generalisable lesson is the one worth taking away from someone else's bug: if a vendor treats geography as a dropdown on the report, ask them what it does to the questions.
Model drift is the smaller half. Your tool drifts too, and nobody publishes when.
His last charge is model drift, and it is the one I would strengthen rather than answer. The engines do move under you, and they move in ways that are not announced as version numbers. OpenAI's own documentation says plainly that with the web search tool available, "the model can choose to search the web or not based on the content of the input prompt." Read that as a measurement person and it is startling: unless you force the tool, the engine decides per call whether to be a retrieval engine or a memory engine. That is a second kind of non-determinism stacked on top of ordinary answer variance, and it is invisible in the output.
We were on the wrong side of that until 18 August 2026. Our ChatGPT probe called the completions endpoint with no tools, so it answered from training weights while every other engine we sell retrieves live. Publishing content is the product, and new content cannot enter a model's weights, so a customer who did exactly what our report told them to do could not shift the reading on the very engine most of their buyers open first. On one customer's probe, the old parametric call named them on none of ten buying questions and the forced-search call named them on three of ten. Treat that as a direction from one brand, not as a rate.
Here is the part the charge misses. In August our own instrument moved four times in nine days, and only one of those four was about an engine at all.
On 12 August a brand-mention check that could force a mention to absent but never rescue one was made symmetric, and the slice of each answer we analyse was widened, because long answers were being cut before the end. That lifted measured rates by roughly 3 percent for reasons sitting entirely on our side of the wire, not in anyone's marketing. On 17 August the geography fix above changed which questions get generated. On 18 August ChatGPT started retrieving. And on 20 August, awareness questions stopped being carried forward into repeat scans, which changes the denominator of the index itself for any brand whose history predates July.
Every one of those moves a customer's number for our reasons. So the rule in our code is relabel, never recompute. No stored rate is rewritten, no scan is re-run, and the dates live in a single list that the trend chart reads, so a customer whose own history spans one of them is told the step may be ours rather than theirs. On two of the four we state no figure for the size of the shift at all, because the direction depends on the individual brand and inventing a number would be the exact defect the list exists to prevent.
This is the question I would put to every vendor in this category, ours included and yours especially, and it is sharper than asking about model versions. On what dates did your own measurement change, and what did each one do to my trend line? Every serious tool has such dates. Almost nobody keeps the list, and a tool that cannot produce one is drawing a continuous line across its own repairs.
The error bars, and the number we compute for every cell and show to nobody
The sharpest line in the thread is not the engineer's, though the number inside it is his. It belongs to a commenter, and it is aimed squarely at us as much as anyone: "17% mention share" with no error bars is just astrology for marketers. Another put the same point without the joke, saying a figure like that is a single sample dressed up as a constant, one roll of the dice.
Both are right, and here is our honest split of it. What we do: paid scans put each question twice, add a third run only where those two contradict each other, and settle the verdict by majority across the runs rather than off a single draw. That is the difference between a number you can compare next month and a record of that afternoon's coin flips. Free scans ask once, and one run establishes presence without supporting a trend, which we say out loud. Two of the six engines are asked once regardless, because they are paid per call and near enough deterministic that extra samples buy nothing.
What we do not do, and I would rather you heard it here. Every multi-sample cell we store carries a confidence value, the fraction of runs in which the brand was actually named, alongside the sample count. It is computed on every scan, it is written to the report, and it is rendered on no screen a customer will ever see. The information exists and the interface swallows it. That is not a defence of the category, it is the same charge landing on us in a slightly more embarrassing form, and it is a rendering job rather than a research problem.
There is a second failure hiding under the error-bars argument that matters more than the bars, because it corrupts the reading rather than blurring it. When an engine returns nothing, that is not a zero. Four separate incidents in our own product share that exact shape: a search provider that replied HTTP 200 and put its quota failure in the payload, Gemini truncating mid-answer because its reasoning consumed the token budget, Perplexity answering 401 for 29 days because our own account had run out of quota, and a forced web search that silently did not run. Each of those would print as "not mentioned" in a tool with two states, and each one would have made the customer look more invisible and our product look more necessary, which is precisely why it needs to be impossible rather than discouraged.
So silence is its own state, excluded from the denominator, and the same principle runs through the smaller decisions. A URL is not prose, so the brand's own domain appearing as a citation link cannot become a mention, which matters more now that retrieval puts links inside answer text. A sentence denying knowledge of a company contains that company's name and is the opposite of a mention, so it is excluded by lookback rather than counted by pattern match. None of this is clever. All of it is the difference between counting what happened and counting what was easy to count.
The engineer's conclusion was that these tools are not useless, that they show directional signals, and that they should stop claiming precision they have not earned. I agree with all three, and I would add the practical test that follows from his own list. Ask a vendor two questions before you ask them anything about features. What does your scan record when an engine returns nothing, and on what dates did your own measurement change? The answers take a competent vendor about thirty seconds. Everyone else will improvise, and you will hear it.
Frequently asked questions
Are AI visibility tools lying about their numbers?
Most of them are reporting an estimate in the typographic costume of a reading, which is a softer thing than lying and does the same damage to your quarter. The six specific failure modes an engineer laid out on the AEO subreddit in June 2026 are all real: front-end scraping bias, model non-determinism, the gap between an API and the consumer app, limited prompt sets and scoring tricks, ignored geography, and drift. We have committed four of those six in our own product and published the dates. The useful question is not whether a tool is imperfect. It is whether the tool tells you where its imperfections are, in writing, before you find them.
What should an AI visibility tool do when an engine returns nothing?
Record it as a third state and drop it from the denominator. Silence is not evidence of absence. Four separate incidents in our own product share that exact shape: a search provider that replied HTTP 200 and put its quota failure in the payload, Gemini truncating mid answer because reasoning tokens ate its budget, Perplexity answering 401 for 29 days because our own account had run out of quota, and a forced web search that did not run. Every one of those would print as not mentioned in a tool with only two states, and every one of them would have made the customer look more invisible and the product look more necessary. An absence is only evidence when the instrument could have registered a presence.
Does geography change AI visibility results?
Yes, and it changes them further upstream than most people assume. OpenAI's own web search tool takes an approximate user location of country, city, region and timezone to refine results. But the bigger lever is which questions get asked at all. We classify each brand into one of four market scopes and generate different buying questions for each, so a wrong scope does not mislabel the report, it spends the entire question budget on the wrong market. We shipped exactly that defect: a brand whose offices span seven countries had four of them kept and a single country inferred from the survivors. Fixed 17 August 2026.
What is model drift and does it break my AI visibility trend line?
Model drift is the engine changing underneath a measurement that assumes it is standing still, and it is the smaller half of the problem. The larger half is that your tool drifts too, and nobody publishes when. Our own measurement changed on four dates in nine days in August 2026: a mention guard made symmetric, a market-scope classifier repaired, a ChatGPT probe switched from recall to retrieval, and awareness questions removed from repeat scans. Our rule is relabel, never recompute. No stored rate is rewritten and no scan is re-run, and a customer whose history spans one of those dates is told the step may be ours. Ask any vendor for the same list of dates.
Is scraping the chat interface more accurate than using the API?
Neither is a control group. Scraping the consumer app carries one account state, one memory, one subscription tier, one session and one geography, and at volume it adds datacentre IP patterns and anti-abuse handling that can bleed into the result. The API removes personalisation and memory but is a third instrument with its own bias rather than a clean baseline, since the app is very likely serving something other than the model snapshot you pinned. We use the API, and we published the measurement rather than the marketing: the same call repeated ten times returns wildly different company sets. The honest position is to name your instrument and its blind spots, not to claim you found the neutral one.
How many times should each question be asked in a scan?
More than once, and the count should be printed on the report. Our free tier asks each question once, which establishes presence and cannot carry a trend line, and we say so on the report. Paid scans run two samples per question and add a third only where those two contradict each other, then settle the verdict by majority across the runs, which is what makes a score repeatable from one month to the next. Two engines that are paid per call and near deterministic are deliberately asked once, because extra samples there buy noise reduction that does not exist.
Can I trust a single AI visibility percentage?
Not as a reading, and not as a comparison between two vendors. You can trust it as a presence check at the extremes. A brand that no engine names on any buying question, run after run, has not been unlucky with the sampling, and that finding survives every objection in this article. What does not survive is a three point move month over month, a ranked position, or any figure quoted to one decimal place without the sample count, the question set and the per engine breakdown beside it. Demand the denominator before you interpret the number.
Sources and further reading
- "Im an experienced software engineer and every AI visibility tool is lying to you!", r/aeo, 30 June 2026. The six charges, the author's follow-up on scraping bias, and the error-bars comment quoted above.
- OpenAI, web search tool documentation. Source of both the line that the model can choose whether to search and the approximate user location parameter of country, city, region and timezone.
- Our own API versus chat interface measurement. The same call sent ten times to each engine, with the distribution of company names it produced.
- Our answer to the snake-oil accusation. The full gradient of mention rate by question type, which is where the denominator argument above comes from.
- The Answer Economy: G2's 2026 AI Search Insight Report, published 15 April 2026, from a survey of 1,076 B2B software buyers in March 2026. Half of them now begin software research with an AI chatbot more often than with Google, up from 29 percent in April 2025, which is why an imperfect measurement still beats no measurement.