Does the ChatGPT API actually search the web? Not unless you turn it on — we found 0 citations across 340 of our own stored responses

We build a tool that measures what AI search says about a brand.

For 340 stored responses, it was measuring what the model remembered.

Not because of a broken parser or a bad key. Because a request body did not contain a search tool, and nobody counted the field that was therefore always empty. This post is the log: what we found in our own code, the exact errors each vendor returns, the three traps that still leave you with zero citations after you switch search on, what it cost, and a check you can run on your own setup in a few minutes.

Grading

1. The thing we got wrong

Two connectors. Both sent a request that looked like this:

gemini    body: { contents, systemInstruction, generationConfig }      ← no tools
openai    body: { model, max_completion_tokens, messages }             ← no tools

Both files also contained a function that reads citations out of the response — groundingMetadata on one side, annotations on the other. Those functions were correct. They were reading a field that never arrived, because we never asked for a search.

Then we counted, for the first time, what was actually in the database ◐:

Zero is a suspicious number. A low citation rate is a finding. Exactly zero across 340 responses is a missing field.

Why it survived so long: every call returned HTTP 200. The parsing code existed and was covered. Nothing logged an error. The only thing missing was data, and no test asserted that data had to exist. The rule we wrote for ourselves afterwards is short: 200 is not evidence. Completion means "we have a response containing a citation URL", not "the code ran."

2. This is the API's default, not a bug we invented

Worth separating two surfaces, because they are genuinely different products:

So if you measure "what AI search says" through an API and you never passed a tool, you measured the model's stored parameters. That answer can be fluent, confident, brand-heavy, and completely disconnected from what is on the web today. It is a legitimate thing to measure. It is not AI search.

We are not the only people to walk into the neighbourhood of this. A developer thread on OpenAI's own community forum opens with the web search tool returning empty annotation arrays and citations that don't match the pages they point at ⬢ (community.openai.com/t/is-web-search-even-working-in-the-api/1150528). Different failure from ours — they had the tool on — but the same shape: the shipped feature returned a response and the sources weren't in it.

🔴 What we are not saying: we did not test any other AI-visibility product. We do not know how anyone else's requests are configured, and we are not going to guess from pricing pages — cheap could mean ungrounded, or it could mean a better contract than ours. If you want to know, ask the vendor the questions in §5. We only know what our own code did.

3. What turning it on actually took

Two vendors, two different walls. Both told us exactly what was wrong, which we only found out by sending the wrong thing first ◐.

OpenAI — Chat Completions has no web search. Both of these are real 400s we received:

tools: [{ type: 'web_search' }]
  → Invalid value: 'web_search'. Supported values are: 'function' and 'custom'.

web_search_options: {}
  → Unknown parameter: 'web_search_options'.

The fix was not a parameter, it was a different endpoint: the Responses API. Moving endpoints renames things, and if you map them wrong your cost accounting silently breaks:

Gemini — the tool key changed generations and the API will tell you. Sending the older key returns a 400 that names the replacement:

google_search_retrieval is not supported. Please use google_search tool instead.

google_search returns 200 with groundingMetadata attached.

⚠ Model versions matter here. Ours were gpt-5.4-mini and gemini-3.5-flash-lite on 2026-07-29. Not every model supports the search tool, and the error text is version-specific. Send the wrong key on purpose once; the message is more reliable than a blog post, including this one.

4. Three traps that still give you zero citations after search is on

This is the part we'd have wanted to read.

Trap 1 — your output token cap eats the citation list. With max_output_tokens at 400, the search ran, the tool call was billed, and we got 0 citations ◐. The list is generated output; cap it and it gets cut. Our free preview path was capped at 512, which would have meant paying full grounded price for a response with no sources in it. We raised the cap only on the grounded path.

Trap 2 — one engine hides the real domains behind a redirect. Gemini returns citation URLs wrapped in vertexaisearch.cloud.google.com/grounding-api-redirect/.... Group your citations by URL host and every source in every answer collapses into one row: Google. Our "top sources" panel was therefore meaningless while looking populated. The real domain is in the chunk's title field; using that split the same data into 14 distinct domains ◐. We did not unwrap the redirects — following third-party URLs is something we've ruled out for ourselves — and anything we can't identify is counted as unknown with the count shown, because quietly assigning it to a domain would make the aggregate wrong in a way nobody can see.

Trap 3 — the money is invisible on one engine and visible on the other. Same question, same output cap, one call each ◐:

Figures are our internal per-response cost in Korean won; the multiple is the part that travels. What matters more than the multiple is where it hides:

• Gemini — input tokens stayed at 37. Google documents that retrieved context "is not charged as input tokens" ⬡; billing attaches to the number of search queries (list price $14 per 1,000 at the time we looked, with a monthly free allowance on the newer models — check current pricing yourself). Read `usage` alone and this cost does not appear at all. One of our requests fired 2 search queries, so "one call" is not "one search."
• OpenAI — the retrieved text arrives as input tokens: 45 → about 13,000 on one call ◐. Token cost jumps, and a per-tool-call charge lands on top (list $10 per 1,000).

Our own cost function multiplied tokens and nothing else, so on the grounded path it was missing 65–90% of the real spend depending on the engine ◐. If you are budgeting AI-search measurement from token math, check whether your engine bills search as tokens, as queries, or both.

5. Check your own setup

Whether you built it or bought it. None of this requires our product.

1. Count citations across every response you've stored. Not the average — the count of responses with at least one. If that is exactly zero over hundreds of responses, stop reading dashboards and go look at a request body.
2. Find the search-usage counter in a raw response. tool_usage.web_search.num_requests on OpenAI, groundingMetadata.webSearchQueries on Gemini. Absent or zero means no search happened, whatever the answer text sounds like.
3. Look at the bill for a search line item. If your cost per question looks like pure token cost, search is probably off. Grounded responses are not subtle: ours went up about 11-fold.
4. Check your output token cap. Search can run, bill you, and still return no sources if the cap truncates them.
5. Ask something that cannot be answered from memory — a question about this week. A confident, sourceless answer is memory.
6. Write down which surface your number came from. API-with-search, API-without-search, and the consumer app are three different measurements. Averaging them produces a number with no referent.

If you're buying instead of building, three questions worth asking a vendor: which surface do you query; can I export a raw response including its citation URLs; is search on for the number I'm looking at. All three are answerable in one sentence by anyone who knows. We are not implying anything about any specific product — see §2.

6. What this cost us in evidence, not just money

Our product had accumulated a 336-response baseline with search off, while our research observations came from separate scratch scripts that had search on. Two different surfaces. So the baseline cannot act as a control group for the research, and we said so in our own decision log rather than quietly comparing them.

That includes our own public number. We had been reporting a 0/2 exposure baseline for our own site. It was measured on the ungrounded surface. It is not comparable with anything we measure from now on, so the baseline restarts and the old figure stays in the record as what it was.

One honest complication, in the other direction. Earlier we had one manual run comparing the Gemini app against the API on 10 questions, checking whether the brand was named ◐. They agreed on 9 of 10. So at the level of "did the brand appear", memory-vs-search did not blow the comparison apart in that tiny test. Where the difference was total was citations, which the API surface could not produce at all. Both of those are ours, both are small-n, and we can't tell you which effect dominates in general.

7. What we still don't know

• Whether grounded measurement changes our conclusions. We have not re-run the baseline. The honest status is "unmeasured", not "improved."
• Whether app and API agree on citations. We only ever compared brand mentions, n=10, one engine, one run each.
• How many search queries a request fires. We observed 2 on one request. That is an observation, not a distribution.
• Whether a returned citation list is complete. We can detect truncation by our own cap. We cannot tell whether the model omitted sources it used.
• Whether these cost multiples hold elsewhere. One question, one output cap, one call per cell.
• Perplexity. We have zero measurements. The empty cell is information about us, not about Perplexity.

Method

Everything marked ◐ comes from our own connectors, our own database, and our own vendor invoices between 2026-07-26 and 2026-07-29, models gpt-5.4-mini and gemini-3.5-flash-lite. Error strings are quoted from responses we received. We keep raw responses. We do not crawl third-party sites, and we did not test any other vendor's product.

We build peekr, which is why we had 340 stored responses to count in the first place — and why the defect was ours to find. If you run a check from §5 on your own stack and get a surprising answer, we'd like to hear it, including if it contradicts something above.