Check the denominator before you trust an AI metric

The same 1,075 answers gave us two different numbers for the same quantity: 12.7% and 37.7%.

We published the second one. This post is about why the first was wrong, why the wrong version was the comfortable one, and how to run the same check on a number somebody is selling you.

The number we almost shipped

We were measuring how often an AI engine disagrees with itself. The unit is a cell — one run × prompt × engine with two or more samples in it. Inside a cell everything a tool controls is held fixed, so anything that changes is the engine changing its mind.

Two rates came out of that archive:

Nothing about the data changed between those two rows. No re-run, no new answers, no different classifier. One filter.

The full measurement, including why we then took the sentiment score off our dashboard, is in We removed sentiment from our dashboard.

Why the extra rows could not move

Sentiment only exists where the brand appears. The classifier reads the spans that name the brand and labels them. In a cell where the brand was never mentioned, there is no span, so the label comes back neutral every time and stays there.

Those cells are not stable. They have nothing to be stable about. Every one of them votes "agrees with itself" and none of them ever had a vote to cast.

Only 61 of the 314 cells had a mention in them. Averaging over all 314 does not measure a stability rate. It measures how much of your data is empty, and prints the result in a box labelled reliability.

The error has a direction, which is why it survives

This is the part worth carrying to other numbers. Ineligible rows do not pull a metric in a random direction. They pull it toward whatever the empty state defaults to — and the empty state is almost always the reassuring one. No mention, so neutral. No data, so unchanged. Nothing detected, so no problem.

So the mistake makes the product look calmer and the instrument look sharper. Nobody files a bug against a number that came out fine.

We did not catch it by being careful. We caught it because 12.7% was suspiciously close to something else: in the same archive, whether a brand is named at all flips 11.8% of the time. Two quantities with no reason to agree had landed on top of each other. That was the only tell, and it was luck that we noticed it.

What the wrong number would have cost

We were building a chart at the time: visibility on one axis, sentiment on the other, four quadrants. At 12.7% it looked like a defensible thing to draw.

At 37.7% it is not, and a second count from the same archive finished it off — across 1,075 parsed answers, negative was assigned zero times. Neutral 947, positive 128. The vertical axis of that quadrant would have been a two-value scale whose two values disagree with themselves 37.7% of the time. We closed the chart.

The cost of the wrong denominator, in that case, was not an inaccurate number on a screen. It was a screen.

The same question, pointed at a different metric

Denominators decide more than sentiment. In the same archive, whether a brand gets named depends almost entirely on which questions are in the set: four brands were named in 100% of answers to questions containing their own name, and in 4.2% to 37.1% of answers to their category questions ◐.

Both of those are computed correctly. They are answers to different questions, and only the denominator tells you which one you are holding. That measurement is Branded vs unbranded prompts.

The same trap appears wherever a share is reported without saying share of what. We worked through another one on YouTube citation figures, where a published share and our own describe the same engine and cannot be compared, because one is a share of all YouTube citations and the other a share of that engine's citations: Does YouTube help you get cited in AI search?

What we do not know about the published work

The largest measurement of this kind we have read is Kumar 2026, which reports 6.8% for mention flips and 45.5% for sentiment flips across 102 brands and roughly 102,000 responses.

We have not verified how that denominator was built. We did not check, and nothing here says it is wrong — the reason we raise it is that it is exactly the question we are telling you to ask, and it applies to papers as much as to dashboards. What the paper is explicit about is a different limit, stated by the author:

Sentiment is classified by a model on the spans that mention the brand, so the 45.5%-sentiment-flip rate mixes genuine output variance with classifier noise; we have not separated the two.

We cannot separate them either.

How to check a rate you are shown

Four questions, in this order. They take minutes and they work on your own numbers as well as a vendor's.

1. How many rows are in the denominator, and what made a row eligible? If the count is not printed next to the rate, ask for it before anything else.
2. How many of those rows could have produced the other outcome? This is the one that caught us. A stability rate over rows that cannot change is not a stability rate.
3. What happens to a row where the thing does not apply — is it excluded, or recorded as the safe value? Recording it as the safe value is the failure mode, and it usually looks like a sensible default in the code.
4. Which way would the error point? If getting it wrong would flatter the number, assume it has not been checked. That was true of us.

For an AI visibility number specifically, question two has a concrete form: of the cells behind this score, how many contained a mention at all?

What we changed

We report rates with the denominator attached, and a rate computed over zero eligible rows is shown as no data rather than as zero. We store every raw response whole, which is the only reason we could recount the same archive two ways instead of arguing about it. And we removed the sentiment score rather than shipping one we could not defend.

None of that makes our numbers correct. It makes them checkable, including by us, a week later, when the number has stopped being flattering. You can see the shape of it on one URL with peekr, and the honest limit is on the same screen: one question asked once is a demonstration, not a measurement.

Frequently asked questions

What is a denominator error in analytics?

Counting rows that were never capable of producing the outcome you are measuring. They cannot show the effect, so they register as agreement, absence or zero and dilute the rate. In our case, cells where a brand was never mentioned were counted in a sentiment stability rate, moving it from 37.7% to 12.7% ◐.

Why does my sentiment score look so stable?

Two candidates worth separating before you celebrate. Either the label really is stable, or most of the rows behind it never contained a mention to classify. Ask what fraction of the denominator had a mention in it: in our archive only 61 of 314 cells did ◐.

Should sentiment be measured only on answers that mention the brand?

That is the only place it exists, so yes — but the honest version of that choice is that it shrinks the sample hard. Ours fell from 314 cells to 61. A vendor doing this correctly has a much smaller n behind its sentiment number than behind its mention number, and should be showing you both counts.

What should I ask a vendor about their metrics?

Three things: the denominator behind each rate, how many of those rows could have produced a different outcome, and how many samples per question the number is built on. If a vendor stores raw responses, all three are answerable in an afternoon. If they cannot answer, they are not storing enough to check their own numbers.

Does this mean AI visibility metrics are unreliable?

That is broader than anything we measured. What we found is one specific error in our own pipeline, caught by accident, in the direction that made us look better. We have not audited anyone else's denominators and are not claiming they have the same problem.

Method. Our own stored measurements, read 28 August 2026: 1,075 parsed answers across 11 brands, two engines (ChatGPT and Gemini), three samples per prompt, raw responses retained. A cell is one run × prompt × engine with two or more samples. Mention flips: cells where brand_mentioned was not constant, 314 cells, 11.8%. Sentiment flips over all 314 cells: 12.7%. Sentiment flips over the 61 cells carrying two or more mentions: 37.7%. Sentiment labels are assigned by an LLM parser over the spans naming the brand; across the 1,075 answers it assigned neutral 947 times, positive 128 and negative 0.

Limits. One vendor's pipeline, our own classifier, no control group, and no way to separate classifier disagreement from genuine engine variance. Eleven brands is small and the corrected sentiment denominator is 61 cells, which is small enough that the 37.7% itself carries a wide interval we have not computed. This post documents one instance of one error found in our own work; it is not an audit of anyone else's, and finding this one does not mean we have found the others.

Source. Kumar 2026, arXiv:2606.20065 — 102 brands, ~102k responses, vendor-collected data, no control group. We have not inspected that dataset and did not verify its denominator.