Measurement

Does an AI engine's API return the same answer your buyer sees?

By Arnav Mukherjee, founder of TofuBofu · July 31, 2026

Updated 15 October 2026 with an independent study, published 11 October, that records the ChatGPT app's own searches and compares them with the API. Jump to it.

TL;DR

No, not identically. The chat app wraps the model in a product layer the API does not have: system prompt, memory, its own web-search rules, model routing and interface experiments. That is a real difference and any tool measuring through either surface should say which one. An October 2026 study of the ChatGPT app found the API and Google together recovered about a quarter of the exact pages it cited, and about half of the websites.

But it is not the biggest source of error. We sent one fixed prompt ten times through the API with nothing changed, and on ChatGPT and Claude not a single company appeared in all ten runs. The surface question is worth asking. The sampling question decides whether the number means anything.

When we built our scanner we had to make a choice that every tool in this category makes and almost none of them disclose: which surface do you measure? You can talk to an engine through its API, or you can drive the logged-in chat interface the way a person does. We chose APIs, for reasons I will lay out honestly, including the ones that are inconvenient for us.

It is a fair thing to be challenged on. If your buyer is typing into the ChatGPT app and our scanner is talking to the ChatGPT API, are we measuring your buyer's reality or a laboratory version of it?

I have been asked this in sales conversations and the honest answer is more interesting than either "yes, identical" or "no, useless." So this post is what actually differs, what we measured ourselves, and how to read any visibility number once you know which surface produced it.

What product layer does the API not have?

The mental model that clears this up: the API gives you the model. The app gives you a product built on the model. The model is the same family in both cases, but the product wraps it in five layers that change what comes back.

The API is stateless by default: every call is a stranger walking in cold. This is the single largest genuine gap between the surfaces, and it cuts in an interesting direction, because a stateless call is a decent proxy for the buyer who has never heard of you, which is exactly the buyer you are trying to reach.

TWO PATHS TO THE SAME MODEL API call Chat app nothing in between system prompt memory + personalization web-search rules model routing interface experiments the same model family run-to-run variance hits both paths

That is a real list and I am not going to pretend otherwise. Anyone telling you the API is a perfect mirror of the consumer product has not looked closely. But before concluding that interface measurement is therefore the honest choice, there is a bigger number underneath all of this.

What did we measure when we sent the same call ten times?

Hold the surface constant. Use the API, where everything is under our control and reproducible. Send one fixed question. Change nothing at all. Send it ten times, and count how much the set of recommended companies moves.

Engine Distinct firms In all 10 runs In 1 run only Mean overlap
ChatGPT610450.20
Claude12201000.09
Gemini471300.23

TofuBofu first-party test, July 2026. Fixed prompt: "What are the best managed IT services providers for a mid-sized company in Canada?" 10 identical API calls per engine, default parameters, no system prompt. Mean overlap is average Jaccard similarity between the company sets of any two runs.

Ten identical ChatGPT calls named 61 different companies, and none of them appeared in all ten. Claude named 122, none in all ten, and 100 of those appeared exactly once. Gemini was the steadiest and still only had one company survive every run.

This reframes the whole surface debate. If you scrape a buyer's chat window once and we call an API once, we have both taken one draw from a wide distribution. The difference between our two draws is dominated by variance, not by the product layer. Arguing about surface fidelity while sampling once is straightening the picture frames in a house with no foundation.

The same effect shows up in the aggregate. Across 4,463 question-by-engine observations from 75 of our scans, we looked at every question where at least one engine named the company. In 43% of those cases the engines split: some named the company, others did not, on the identical question. There is no single answer to faithfully reproduce, because the engines do not agree with each other and do not agree with themselves.

What happened when researchers recorded the ChatGPT app itself?

Until October 2026 this argument rested on reasoning. Now there is a measurement. Five researchers from Xi'an Jiaotong University and the National University of Singapore recorded what the ChatGPT app does behind the screen: the searches it runs, the pages it pulls back and the pages it cites. They captured those traces from their own logged-in sessions, then asked the same questions through the API and through Google, and compared.

Share of what the ChatGPT app cited that Google and the API together also found
Level of match GPT-5.3 Instant GPT-5.5 Instant
The exact page28.3%26.2%
The same website50.3%48.7%
The same named entity50.7%45.1%

Jiang, Zhang, Xing, Chang and Xiao, arXiv:2610.15382, 11 October 2026, Tables 4 and 5. "Entity" means a company, product or other name found in the title and snippet of a cited page. The authors' cited set also counts supporting pages the app attaches to a reference without always showing them.

In the authors' words: "Even when combined, they recover only about one quarter of cited URLs." At the level of the website, or the name, they recover about half. So a tool that hands you a list of exact ChatGPT URLs and calls it "what your buyer sees" is claiming more than anyone has shown an API can deliver.

Three more findings from the same paper back the argument above:

Two limits to keep in view. These were everyday questions, not B2B vendor shortlists. And the authors state that their entity results do not support separate conclusions about brands. The study shows that coarse matches hold up better than exact pages. It does not prove that "was my company named" transfers from the API to the app at any particular rate. We have not seen that number published.

Here is what it changes in how we describe our own product. Our ChatGPT results come from the API with web search switched on. The headline we report is whether ChatGPT named you, which sits nearer the coarse level this study found more robust than the exact page. The sources in our report are pages the API cited. Read them as "these are the sites ChatGPT draws on for this question", not "these are the exact links your buyer clicked". On the evidence above, the website is the claim that holds and the exact page is the one that does not.

See the per-engine spread on your own questions

A free scan samples five AI engines and reports each one separately, with the full verbatim answer per engine so you can check the result by hand.

Run your free scan

Engines behave differently enough that blending them is malpractice

While we had the instrumentation open, we logged response times across every engine call our scanner made. Latency is a crude instrument, but it is a revealing one, because it tells you roughly how much retrieval work is happening before an answer comes back.

Engine Median response Presence rate n (observations)
Google AI Overviews2.5s3.1%326
Bing Copilot2.5s4.6%327
Claude7.2s12.4%1,206
ChatGPT8.0s13.3%1,259
Perplexity8.4s15.1%866
Gemini13.4s11.5%479

TofuBofu first-party data. Median response time from 1,558 timed engine calls; presence rate from 4,463 question-by-engine observations across 75 scans, June 24 to July 27, 2026. Engine coverage differs because not every scan ran every engine.

The two search-derived surfaces come back in about two and a half seconds and name the company least often, at 3.1% and 4.6%. The conversational engines take three to five times longer and name companies three to five times more often. That is not a coincidence: a snippet-shaped answer built from a results page has room for fewer names than a generated paragraph that is happy to list eight vendors.

A tool that averages those six numbers into one score is averaging a 5x spread. Your blended number goes up and you cannot tell whether you gained on Perplexity or lost on AI Overviews, which are completely different problems with completely different fixes. This is why we report per engine and why I would push back on any tool that will not.

Why did we choose the API, including the inconvenient reasons?

Four reasons, in order of how much they actually mattered to the decision.

  1. Sampling beats fidelity, and the API is what makes sampling affordable. Given the variance numbers above, the single highest-leverage thing a measurement system can do is ask more times. API calls are cheap and parallel, so you can afford repetition. Driving logged-in interfaces is slow and expensive, so in practice it pushes you toward smaller samples. A high-fidelity single draw loses to a lower-fidelity average of many draws when variance is this large.
  2. Reproducibility. We can tell you the model, the parameters, the exact prompt and the date. You can re-run it. A scraped interface result depends on an account's memory, that account's experiment bucket and the session's history, none of which you can hold constant or hand to a customer who wants to check our work.
  3. The stateless call matches the buyer we care about. Personalization is the biggest genuine gap, and it matters most for buyers who already know you. The commercially urgent question for most B2B companies is whether a stranger with no history gets told your name. A cold API call is a reasonable model of exactly that person.
  4. It is the permitted path. APIs are the sanctioned interface with published terms and stable behaviour. Automated collection from consumer interfaces generally runs against those products' terms of service, which makes it both an ethical question and a durability question: measurement infrastructure that depends on not being blocked is infrastructure that eventually breaks.

And the honest counterweight, because this is a real trade: we do not see personalization, we do not see interface-only features like shopping modules and citation chips, and we do not capture product-side system instructions.

If your question is "what does this specific returning customer see in their app," we are the wrong instrument and I would say so on a call. If your question is "when a buyer who has never heard of us asks for a recommendation, do we get named," we are measuring precisely that, repeatedly, on five engines.

How do you read anyone's visibility number now?

Four questions, in the order that matters:

1. How many times was each question asked?

This is the first question, not the fourth, because it dominates everything else. One pass per question is a coin flip dressed as a metric. If the answer is a single run, the surface debate is irrelevant and the number is mostly noise.

2. Which surface, and does the tool say so unprompted?

Both API and interface measurement are defensible. Silence about which one is being used is not. A tool confident in its methodology will tell you, and will also tell you what its choice fails to capture.

3. Is the question set frozen between scans?

If the questions change from scan to scan, the denominator changed and the trend line is fiction. You cannot compare 8% on one set of questions to 12% on a different set and call the difference progress.

4. Is it reported per engine?

Presence ranged from 3.1% to 15.1% across engines in our data on the same companies and questions. One blended number hides which engine you are losing, which is the only part you can act on.

The API and the app are genuinely different surfaces, and anyone who tells you otherwise is hand-waving. But the surface question turns out to be the second question. The first one is how many times you asked, because on two of the three engines we tested, ten identical calls produced no company that appeared every time.

Get sampling right and the surface choice becomes a disclosed methodological trade-off with honest costs on both sides. Get sampling wrong and it does not matter which window you were looking through.

Frequently asked questions

Does the ChatGPT API give the same answer as the ChatGPT app?

Not identically, no. The consumer app wraps the model in a product: a proprietary system prompt, saved memory and personalization, its own rules about when to run a web search, routing between model variants, and interface experiments that vary between users. The API gives you the model with none of that product layer unless you add it yourself. Both are talking to the same underlying model family, so the substance of an answer is usually similar, but the specific companies named in a recommendation can differ because retrieval and context differ. Treat them as related surfaces rather than one surface. An independent measurement of the gap, published in October 2026, recorded the ChatGPT app's own searches and found that the API and Google together recovered about a quarter of the exact pages the app cited, and about half of the websites.

Which surface should an AI visibility tool measure?

Both have real trade-offs. The API is reproducible, permitted by the engine's terms, parameterized, and cheap enough to sample many times, which is what beats variance. The logged-in interface is closer to a specific buyer's lived experience but is personalized per account, unstable across interface experiments, expensive to collect at scale, and typically obtained in ways the engine's terms discourage. Neither is the true answer, because there is no single true answer to reproduce. What matters more than surface choice is sampling depth and whether results are reported per engine.

Why do AI engines give different answers to the same question?

Because generation is probabilistic and retrieval is dynamic. We sent one fixed prompt to each engine ten times through the API with default parameters and changed nothing between calls. ChatGPT named 61 distinct companies across the ten runs with none appearing in all ten. Claude named 122 with none appearing in all ten. Gemini named 47 with one appearing in all ten. Two random Claude runs shared about 9 percent of their answer. The engine is sampling from a distribution each time, and when the engine also runs a live web search, the retrieved set can differ between calls too.

Is scraping the chat interface better than using the API?

It is different, not automatically better. Interface data captures the product layer that the API omits, which is genuinely useful. But it inherits problems the API does not have: answers are shaped by that account's memory and history, interface experiments mean two accounts can see different surfaces on the same day, collection is slow and expensive so sample sizes tend to be smaller, and it generally runs against the engine's terms of service, which makes it fragile. A small interface sample is not obviously more trustworthy than a large API sample, because the dominant source of error in both is run-to-run variance.

What does the API genuinely fail to capture?

Four things worth naming. Personalization, because a returning buyer with saved memory may get an answer shaped by their history. Interface-level features like shopping surfaces, citation chips and inline links that exist in the product but not the raw model response. Product-side system instructions that influence tone and hedging. And any interface experiment being run on a subset of users. These are real limitations. They affect the texture of an answer more than they affect whether a mid-market vendor gets named for a category question, which is the thing most B2B companies are trying to measure.

How many times should a question be sampled to be trustworthy?

More than once, and the right number depends on the engine. Engines that lean on live retrieval tend to be more repeatable than engines that generate from model knowledge, so they need fewer runs to stabilise. In our own testing, a single run is clearly insufficient on every engine we measured, since no company appeared in all ten runs on two of the three engines tested. Sampling several times and reporting how often a company appeared, rather than whether it appeared, is what turns a coin flip into a measurement.

Does this mean AI visibility numbers are worthless?

No, it means they are estimates and should be presented as estimates. A per-engine presence rate built from repeated sampling of a frozen question set is a genuinely useful number: it is comparable over time, it points at specific questions and engines to fix, and it is checkable by hand. What is not useful is a single blended score from one pass per question, presented without a denominator or a surface disclosure. The fix is disclosure and sampling, not abandoning measurement.

Sources and further reading

Related reading

Is AEO and GEO just snake oil?
Your AI visibility score says 36%. ChatGPT never named you.
How often should I check my AI visibility?

Arnav Mukherjee

Founder, TofuBofu