Measurement
How to check whether your AI visibility report is telling you the truth
By Arnav Mukherjee, founder of TofuBofu · August 10, 2026
A report told me to add a "Solution" schema to one of my pages. There is no Solution type in schema.org. A little further down, the same report said my pricing page was missing Offer markup. That page has three Offer blocks in it, and I could see them by pressing view source.
The report was produced by my own product, which is what made me stop rather than shrug. We audited six of our own reports line by line, recomputing every number from the raw data and checking every recommendation against the live sites. What came back was not what I expected, and it is the reason this post exists in the shape it does.
Every measurement was correct. All 489 answered question-and-engine cells across those six reports matched their own stored evidence, in both directions, with no exceptions. And every single fabrication was in the advice underneath. That split is not a quirk of our build. It is structural, it applies to every tool in this category including the one you are paying for, and once you know where the seam is you can audit any report in about twenty minutes. Here is how.
The one thing to understand before you check anything
Your report is two products stapled together, and they have opposite failure modes.
The measurement layer asks engines your buying questions, stores what comes back, and counts whether your name is in the text. It is arithmetic over evidence. When it breaks it breaks loudly, because every number it produces can be held up against the answer it came from and compared.
The generation layer takes a summary of your site and your results, hands it to a language model, and asks what you should do next. It is the part that writes your recommendations, your content briefs, your technical fixes and your executive summary. When it breaks it breaks silently, because a fabricated recommendation is fluent, specific, confidently worded and completely indistinguishable in tone from a true one.
Notice the asymmetry that creates. The measurement is the part vendors put on their landing pages. The generation is the part you actually act on when you sit down on Monday to decide what to change. The layer with the most influence over your quarter is the layer with the least verification behind it, and almost no report tells you where one ends and the other begins.
Where any claim in your report was born, and what that means for checking it
Part one: can you see the evidence at all
Two checks, and if the first one fails you can stop, because nothing after it is verifiable.
Open one question and read the engine's full answer
Pick any question in the report, ideally one where you scored badly, and find the verbatim response from each engine. Not a summary, not a score, not a 200-character preview. The whole answer as the engine produced it.
Pass: Full text, per engine, per question, expandable in the interface.
Fail: A number with no text behind it, or a truncated snippet. Truncation matters more than it sounds: your brand can be named in the last paragraph of a long answer, after the preview cuts off, so a snippet cannot support an absence claim. If you cannot read the answer, the product's central claim about you is unfalsifiable and every check below is unavailable to you.
Test one absence verdict against the text, then test a mention verdict
Take a cell that says an engine did not name you and search that answer for your company name. Then search for the variants: with and without the legal suffix, with and without punctuation, the domain form, the abbreviation your buyers use, the way the name breaks if it contains a dot or an ampersand. Then do the reverse on a cell that says you were named, and confirm your name is genuinely in that text.
Pass: Both directions correct. Across the six reports we audited, all 489 answered cells were right in both directions, so this is an achievable bar rather than a demanding one.
Fail: A false negative is the expensive one, because it undercounts you everywhere and silently. If the tool stores 'Acme' and misses 'Acme Technologies, Inc.', that error repeats on every question and every engine, and your entire score is wrong in a direction that makes the product look more necessary.
Part two: is the measurement sound
These four are questions to put to the vendor rather than things you can grep, and the answers are short. A vendor who has thought about measurement will answer all four immediately. A vendor who has not will improvise, and you will hear it.
Ask what happens when an engine returns nothing
Engines time out, rate-limit, refuse questions and occasionally return empty responses. Ask directly: when that happens, what does the scan record, and what does that do to my score?
Pass: A distinct state, excluded from the denominator. No answer is not evidence of absence in either direction, and treating it as its own outcome is the only honest handling.
Fail: Silence scored as an absence. This is one of the most common silent corruptions in the category because a failed call and a real absence look identical in a dashboard. We shipped this bug ourselves: a search provider was answering quota and auth failures with an HTTP 200 and an error payload, our code never checked for it, and two engines' results were quietly recorded as 'not mentioned' until we caught it. If every cell in the report is a binary yes or no, ask what an API failure becomes.
Get the per-engine breakdown out of the headline number
Whatever single score sits at the top, ask to see it decomposed by engine before you interpret it.
Pass: A per-engine table you can read, where a zero on one engine survives as a zero.
Fail: A blend that averages a hard zero away. Engines differ structurally: some retrieve live sources before answering and some compose from training data, so being invisible on one and present on another is a specific, actionable finding rather than noise to be smoothed. A blended number is the one output guaranteed to hide the most useful thing in the report.
Ask whether the questions stay the same between scans
Ask what happens to the question set on next month's scan. This is the check almost nobody runs and it invalidates more trend lines than everything else on this list combined.
Pass: The set is reused, with additions only when your plan's question budget grows.
Fail: The set regenerates each time. Then this month and last month are two different tests and the line between them means nothing. We found this in our own data: two consecutive scans of the same brand shared zero questions, and an eight percent result followed by a zero looked like a collapse when it was two unrelated measurements. If a vendor cannot answer this one, do not read anything into your trend.
Ask how many times each question was asked
Models are non-deterministic. Ask the same engine the same question twice and you can get two different shortlists, with nothing having changed on your side.
Pass: Multiple samples per question per engine, with the verdict taken across them.
Fail: A single run per question, reported as a measurement. That is a coin flip wearing a percentage sign. It matters most on the questions where you are borderline, which are exactly the questions worth acting on.
Check 5 has a bigger sibling worth pulling on if the vendor's answer is vague. Beyond whether the questions stay the same, ask where they came from in the first place. We ran that test on ourselves and the result was uncomfortable enough to write up separately, in where do the questions in your AI visibility report come from.
Run these checks on a report that lets you
A free scan shows every engine's full verbatim answer to every buying question, so you can check the verdict against the evidence yourself.
Get your free auditPart three: is the advice connected to anything
This is where the fabrications live, and the checks are all mechanical. Not one of them requires judgement about whether a recommendation is good. They only ask whether it is connected to a fact.
Take one technical recommendation and check your own page source
Pick a recommendation that says something is missing from a page. Open the page, view source, search for it. Thirty seconds.
Pass: The thing really is missing.
Fail: It is already there. This catches two very different problems at once and it is worth knowing which one you have hit. Either the model invented a requirement, or the crawler could not see what exists. The second is sneakier: ours read only the top-level type of each structured-data block, so an Offer nested inside a SoftwareApplication's offers property was invisible to it. Our pricing page had three of them, the crawler reported none, and the model was truthfully told Offer was absent and truthfully wrote it up. Two correct components, one false statement about a customer's own site.
Check every recommended schema type against schema.org
Take each structured-data type the report tells you to add and search for it at schema.org. Ten seconds each.
Pass: Every type exists in the vocabulary.
Fail: An invented type. Two of the six reports we audited contained one: a 'Solution' schema on ours and a 'CaseStudy' schema on another. Neither exists. A model asked to name a missing structured-data type will produce something that reads like a schema.org type whether or not one is there, and this is the single cheapest fabrication to catch, which makes finding one a strong signal about everything else in that section.
Trace one content recommendation back to a scanned question
Every suggested article, comparison page or FAQ should point at a buying question the tool actually asked an engine. Take the target query printed on the recommendation and find it in the list of questions the scan ran.
Pass: It is in the list, and ideally the report shows you how you performed on it.
Fail: It is not there. Then the recommendation is not connected to a measurement, whatever it looks like on the page. This was our worst finding and it is invisible unless you deliberately cross-reference: on one report, two of three content briefs named target queries the scan had never run, printed under a TARGET QUERY heading as though measured, and four of five action items cited queries that were not in the scan. Nothing about the presentation distinguished them from the one brief that was genuinely anchored.
Demand arithmetic behind any forward-looking number
Anything of the form 'this will fix N queries' or 'this is worth X percent' should have a computation behind it that someone can describe in one sentence. Ask which queries, by name.
Pass: A list you can check, or the number is simply absent.
Fail: A plausible integer. One of our audited reports carried a quick win claiming it would fix five queries and there was no computation anywhere behind the five. Forward-looking counts are the easiest thing in the whole document for a generation step to invent, because they look like output from a model of cause and effect that does not exist. The honest version of that field is either arithmetic or empty space, and we chose empty space rather than a better guess.
Find out whether the report can ever say there is nothing to fix
Ask the vendor: what does a report look like for a company whose technical foundation is already clean? Or look at your own report and count. If it always finds five things, or always ten, ask why the number is so round.
Pass: Zero findings is a possible and clearly stated outcome, and a clean foundation is reported as a good result rather than an empty section.
Fail: A fixed quota. This is the upstream cause of most of the rest of this list. Our diagnosis prompt used to ask the model for a set number of findings, and a model told to produce five gaps on a site with two will produce five. The three that do not exist have to come from somewhere, and they come out sounding exactly like the two that do.
Three things you cannot check, and how to hold them
A test that claims to verify everything is its own kind of dishonesty. Some of what is in your report is genuinely unverifiable, and knowing which parts is as useful as the checks above.
Sentiment scores. These are a model's read of tone, and on a small or young brand they partly measure the engine making things up about a name it does not know. Our own report scored positive sentiment on an excerpt in which the engine described a company at the wrong domain entirely and called it something we are not. The number was not wrong about the text. The text was wrong about us. Read sentiment as directional at best, and always read the excerpt behind it.
Competitor sets pulled out of answers. Extracting company names from prose is a model task, and models do not know what counts as a competitor. Ours has returned analyst firms and a research repository in a competitor list, because they appeared in the answer as named entities. Treat the list as a starting point to prune rather than a finding, and be suspicious of any share-of-voice number computed on top of an unpruned one.
Anything about what will happen next. Nobody in this category has a validated causal model linking a specific fix to a specific change in an AI answer, including us. Timeline and impact claims are informed guesses. That is fine as long as they are labelled as guesses, and it is the reason check 10 exists.
What to do with the results
Score it the way you would score a reference call. Checks 1 and 2 are pass or leave: if you cannot read the evidence, or the evidence contradicts the verdict, nothing else in the document is worth your attention. Checks 3 to 6 tell you whether the number is a measurement or a vibe. Checks 7 to 11 tell you how much of the advice you can act on without verifying each line yourself.
A tool can fail several of the generation checks and still be worth keeping, as long as it passes the measurement ones. Advice you have to verify is annoying. A score you cannot verify is worthless, and worse than worthless if you reorganise a quarter around it.
If you find fabrications, the useful thing to ask a vendor is not "why did this happen", because the answer is always that a model wrote it. Ask what runs between the model and the report. The answer you want is a deterministic validation pass with no AI in it: recommended schema types checked against the real vocabulary, gap pages checked against what the crawler actually saw, target queries checked against the questions the scan actually ran, and anything that fails deleted rather than softened. That is what we built after our audit, and each removal writes a line to the log so the invention rate is a number we watch rather than something a founder discovers on a Tuesday.
There is an obvious irony in a tool that exists because AI engines describe companies confidently and incorrectly, describing a company confidently and incorrectly. The difference is that a fabricated schema type is mechanically checkable, so we can refuse to print it, whereas an engine telling your buyer that a competitor is the best firm in Columbus is not checkable by anyone at the moment of reading. That asymmetry is the whole job: wherever a claim can be verified deterministically, verify it and delete what fails. Never ask a model to grade its own homework.
The reason to spend twenty minutes on this at all is what the report is now standing in for. In G2's 2026 buyer research the share of B2B buyers starting vendor research on an AI chatbot had risen to 51 percent from 29 percent, and 69 percent said an AI had changed which vendor they picked. So a document describing that moment is a document people restructure a quarter of work around. Checking it is the cheapest thing you will do all quarter, and if it survives, you get to trust it properly, which is the actual prize.
None of this changes where the work lies. Being crawlable and cleanly marked up is the floor: necessary, cheap, and nowhere near sufficient, because a retrieved answer is decided on whether independent sources corroborate you rather than on your markup. Auditing the report sharpens what you know about the floor. Getting named is still a separate layer, and it is one you can win even when your Google rank does not move.
Frequently asked questions
How do I know if my AI visibility report is accurate?
Check the falsifiable claims, and check them in the right order. Start with one 'you were not mentioned' verdict: open that question's stored answer and search the text for your company name and its variants. Then take one technical recommendation and search your own page source for the thing it says is missing. Then take one content recommendation and confirm its target query is actually in the list of questions the scan ran. Those three checks take about ten minutes and they separate a tool that measured something from a tool that generated something. A report that cannot show you the underlying answer text fails the first check by design, and nothing after it can be verified.
What is the difference between the measurement and the recommendation in an AI visibility report?
They are two different products in one document and they fail in opposite ways. The measurement layer asks engines your buying questions, stores the answers, and counts whether you appear, so every number it produces traces back to text you can read. The generation layer hands a summary of your site and your results to a language model and prints what it writes back. The first fails loudly, because a broken count is visible against its own evidence. The second fails silently, because a fabricated recommendation is fluent, specific and reads exactly like a true one. Most of what a customer acts on sits in the second layer, which is the wrong way round.
Why does my AI visibility report say a page is missing schema it already has?
Usually the crawler rather than the model. Many crawlers read only the top-level type of each structured-data block, so an Offer nested inside a SoftwareApplication's offers property is invisible to them. We hit this on our own pricing page: it carries three Offer blocks, our crawler reported none, the model was truthfully told Offer was absent, and it truthfully wrote that up as a gap. Nobody in the chain misbehaved and the customer still got a false statement about their own site. This is why the check is view-source rather than argue-with-the-vendor, and why a single false gap should make you re-read the whole technical section rather than just that line.
Should an AI visibility tool show me the full AI answer?
Yes, and this is the check to run first because everything else depends on it. The central claim of the entire category is that a given engine did or did not name you on a given question. If the tool will not show you the verbatim answer behind that claim, the claim is unfalsifiable and you are being asked to take the product's core output on faith. A truncated snippet is not enough either, because your brand can appear late in a long answer, after the snippet ends. Full text, per engine, per question, is the minimum standard for anything you are paying for.
What happens if an AI engine returns no answer during a scan?
It should be recorded as its own state and dropped from the denominator, not counted as an absence. An engine that timed out, rate-limited you, refused the question or returned an empty response has produced no evidence about your brand in either direction. Scoring that as 'not mentioned' quietly drags your number down for reasons that have nothing to do with your visibility, and it is one of the most common silent corruptions in this category because it looks identical to a real absence in a dashboard. Ask your vendor directly what a failed engine call does to the score. If every cell in the report is a binary yes or no, the honest answer is usually that it becomes a no.
Do AI visibility tools invent recommendations?
The generation step will invent whenever nothing checks it, and that is a property of generation rather than a claim about any particular vendor. When we audited six of our own reports we found invented schema.org types in two of them, one recommending a 'Solution' type and another a 'CaseStudy' type, neither of which exists in the vocabulary. We also found content briefs pointing at queries the scan had never run and a quick win claiming it would fix five queries with no arithmetic behind the five. Every one of those is mechanically checkable in under a second by something that is not a model. The fix is a deterministic validation pass that deletes what fails, and the buyer-side version of that fix is the checks in this post.
Should the questions in my AI visibility scan change every month?
No, and this is the check most people never think to run. If the tool regenerates its question set on every scan, then this month's score and last month's score came from two different tests, and the line between them is not a trend, it is an artefact. We learned this the hard way: two consecutive scans of the same brand shared zero questions and turned an eight percent result into a zero, which looked like a collapse and was actually two unrelated measurements. A repeat scan should reuse the previous set and only add questions when your plan's budget grows. Ask what happens to the set next month before you read anything into the direction of your score.
Sources and further reading
- The schema.org type hierarchy: the published vocabulary, and the list an invented type such as Solution or CaseStudy fails against. Check 8 is a search on this page.
- Google, Introduction to structured data: how nested structured data is expressed, which is exactly what a top-level-only extractor misses.
- Lewis et al., Retrieval-Augmented Generation (arXiv:2005.11401): the mechanism behind why grounding a model in retrieved documents produces more factual output than generating from parameters alone, which is the reason the generation layer needs grounding of its own.
- TofuBofu, AI visibility for managed IT providers, 2026: our published first-party research, including the method used to verify every company an engine named.
- G2 2026 B2B buyer research: 51 percent begin vendor research on an AI chatbot, up from 29 percent, and 69 percent switched vendor based on AI.