Does ChatGPT know my brand? Asking it directly can't tell you — a name we invented scored 90.5% and a real product scored 7.1%

There is a check that almost every guide on AI visibility recommends, and it takes about a minute.

Open a temporary chat, type your own brand name into ChatGPT, and see whether it knows you.

We sell AI-visibility measurement, and we had never tested whether that check works. So we tested it. It doesn't — not in the way people are using it. The number it gives you is mostly a function of how you phrased the question, and we can show you the size of that effect.

Grading

1. The check everyone recommends

This is real, published advice. One guide we read gives six prompts to run against your own brand ⬡, and the first two are:

What do you know about [your brand]?
Is [your brand] a real company, and what do they do?

(skyscale.com.au/blogs/check-if-chatgpt-knows-my-brand, read 2026-08-02.)

The framing around this advice is consistent across the category: use a temporary chat so memory doesn't flatter you, ask cold, and read what comes back. That part is good advice and we repeat it below.

The part we could not find anyone checking — and we looked, see §7 for the closest thing — is whether the answer to "Is X a real company" carries any information about X.

2. How we measured it

The problem with reading the prose. If you ask a model whether a company is real, you get a paragraph. Deciding whether that paragraph counts as a yes is a judgement call, and we would be the ones making it about our own experiment. So we didn't read the prose. We read the number underneath it.

Setup ◐:

Why one call per cell. A logprob is not a sample, it is the distribution the sample would have been drawn from. Asking the same thing three times and averaging the answers would be re-measuring the same number with extra steps. Robustness had to come from somewhere else, so it came from varying the wording — Experiment 2 asks the same thing three ways.

⚠ That is a design assumption and we did not verify it. We did not check whether identical requests return identical logprobs on this endpoint; production inference backends have sources of run-to-run variation that have nothing to do with sampling. If they drift, our single calls carry an error bar we have not measured.

Seven brand strings, chosen to span the range from a household SEO name to a name we believed did not exist at all.

3. Experiment 1 — the leading question passes everything

Prompt: Is X a tool for tracking brand visibility in AI search answers?

The floor of that column is 90.5%, and the floor belongs to a company that does not exist.

Two of the 100.0% rows are worse than uninformative. Trackwell and Beaconly are real businesses that have nothing to do with AI search visibility — the model did not merely fail to discriminate, it asserted something false about two identifiable companies. Which we only found out because we made a mistake.

4. 🔴 We got our own control group wrong

We picked three names we believed were fictional: Trackwell AI, Beaconly, Mentiono. Then we searched for them.

Two of the three were real companies. We opened their sites ⬡:

• Trackwell (trackwell.com) — a software company in Reykjavík, Iceland. Its site describes mobile resource management, fisheries resource management and enterprise software, claims over 200 employees and deployment in 11+ countries, and says it has been doing this for "over three decades." (We did not find an explicit founding year on the homepage.)
• Beaconly (beaconly.app) — a macOS menu bar productivity app, shipping on the Mac App Store, advertising 363+ integrations.

Neither sells anything resembling AI-visibility tracking. Our "fake" control group was one third the size we thought it was.

Why we're telling you this instead of quietly fixing it. Because the mistake is where the finding came from. We set out to catch false positives — a model confirming a company that doesn't exist. That is a mildly interesting result and Experiment 1 got it. What we accidentally built was the harder case: two real companies with thin footprints, sitting in the dataset labelled as fakes. Experiment 2 is only worth reading because they were there.

⚠ Two more disclosures on the same table. Experiment 1 asked about the string Trackwell AI while Experiment 2 asked about Trackwell — different strings, our inconsistency, and the rows are not strictly comparable. And Mentiono returned no search results when we looked; if it exists by the time you read this, that row is void.

5. Experiment 2 — the skeptical question fails real companies

Same model, same day. Three phrasings of the check people are actually told to run:

1  Is X a real, existing software company?
2  Does the company X actually exist?
3  I am considering buying software from X. Is this a legitimate real vendor?

P(Yes) per phrasing, and the mean of the three ◐:

Read the Beaconly row against the Mentiono row. A product you can download today, with a public site and an App Store listing, scored 7.1%. A name we made up scored 3.5%. The gap between "exists and ships" and "does not exist" is four percentage points, and it is the wrong four points for either number to be usable.

And look at Trackwell's first cell. The same 200-employee company is at 62.2% under one phrasing and 100.0% under the other two. Nothing about the company changed between those three calls.

6. What the number is actually tracking

Put our own row from both experiments side by side ◐:

Same brand, same model, same day. 85 percentage points, and the only thing that changed was the question.

That is the whole finding: when you ask a model about a brand with a thin footprint, the output follows the framing of your prompt more than it follows the brand. A leading question gets agreement — including for companies that don't exist. A skeptical question gets denial — including for companies that do.

So a "yes" is not evidence that the model knows you, and a "no" is not evidence that it doesn't.

⚠ What we are not claiming. We did not measure whether these models "know" any of these brands. We have no access to training data and no ground truth to check against. We measured one narrower thing: this particular check does not discriminate, in a category where it is being recommended as a diagnostic.

7. Someone made this argument before us. Here's the difference

Yotpo published "Why You Can't Just Ask ChatGPT About Your Brand" △▲, and it is not a thin post — they ran six question variations across geographies and report that in 83% of question–geo combinations the set of brands changed between runs, and that the top recommendation changed in over 50% of cases. We read their write-up. We did not see raw data and have not replicated it, and they sell in an adjacent space.

Their evidence and ours are about different failures, which is why both are worth having:

8. What this cannot tell you

• One engine. OpenAI only. Gemini, Claude and Perplexity do not expose token probabilities the same way, so we could not run this design on them. Nothing here generalises to those engines and we did not test them.
• Seven brand strings, three phrasings, one call per cell. Small. Treat the direction as the finding, not the decimals.
• `top_logprobs` is capped at 5. If Yes/No had fallen outside the top five candidates for a cell, there would be no value to renormalise. Every cell here produced one; a different phrasing might not.
• Model version is part of the result. gpt-5.4-mini, 2026-08-02. A vendor can change the model under you and these numbers would move with no visible cause. Our own database still stores the engine but not the model version ◐ — a hole we've written about before and haven't closed.
• Forced binary. Renormalising over Yes/No discards the mass the model put on everything else — hedges, questions, refusals. If a model wanted to say "I'm not sure," our method converted that into a position it did not take.
• Ungrounded. No search tool was attached, by construction. This measures stored parameters, not AI search. Numbers from the two surfaces are not comparable and should never be averaged.
• We did not test any product. Every brand in the tables above is a string we sent to a model. The counts and probabilities say nothing about whether any of these products are good, and we did not use, review or benchmark any of them.

9. What to do instead — no tool required

You still want to know where you stand. The method that survives everything above is duller and takes an hour.

1. Stop asking about yourself. "Is [brand] a real company" is the question we just showed you can't use. Ask what a buyer asks: "What's the best [category] for [situation]?"
2. Ask it many times, in a fresh temporary chat each time. One answer is a screenshot; a rate across ten is a finding. Log out — a signed-in session carries your own history back to you.
3. Write down every other brand named, not just whether you appeared. The names that recur are the ones the model reaches for by default in your category, and they're the pages worth going and reading.
4. Record whether web search was on. In the consumer apps it usually is. Through an API it usually isn't unless you turned it on. Never mix the two into one number.
5. Report a rate with its denominator. "2 of 10" is honest. "ChatGPT doesn't know us" is not something a single answer can establish.

We did exactly this at larger scale on one brand — 24 questions, 2 engines, 7 samples each, 336 stored answers — and wrote up what came out, including the part where the headline percentage turned out to be an artifact of the question list: How do I find out which competitors AI recommends instead of me?

Method

All figures marked ◐ come from calls we made on 2026-08-02 to OpenAI gpt-5.4-mini via /v1/chat/completions with logprobs: true, top_logprobs: 5, max_completion_tokens: 16, no search tool attached. P(Yes) is the probability mass of first-token candidates beginning Yes, renormalised against those beginning No. One call per brand × phrasing. Raw responses retained.

Facts about Trackwell and Beaconly come from opening trackwell.com and beaconly.app ourselves on 2026-08-02 and reading what those companies say about themselves. We do not crawl third-party sites. We did not test, review or benchmark any product named in this post, and no figure here is a judgement about product quality.

We build peekr, which is the reason we had a strong incentive to publish the opposite of this result — a one-minute check that worked would be a fine top-of-funnel demo. It doesn't work, so this is the post instead.

If you can run this design on an engine that exposes logprobs and get a different answer, we'd like to see it — including if it contradicts us.