LLM Hallucination Rates: What Counts as a Wrong Answer?

By Yuki Yu

Give a support assistant a successful API response and ask two questions. “How many tokens did this request use?” has an answer in the record. “How much did my account pay?” may require a billing entry. The assistant can handle the first question perfectly and invent a plausible answer to the second. A buyer looking at a single accuracy score has little idea how often that will happen.

An LLM hallucination rate becomes interpretable when you know which answers count as errors, what goes in the denominator and what happens when the model declines to answer. Two models can get nearly the same share of questions right while sending very different numbers of wrong answers to customers.

Read first. Define the decision a model will support and what a successful answer looks like in that setting. Token360’s model evaluation framework.

How can similar accuracy hide a large difference in wrong answers?

Accuracy counts correct answers across the test. The remaining responses can include both errors and abstentions. In a September 2025 SimpleQA comparison, OpenAI reported that gpt-5-thinking-mini answered 22% correctly and 26% incorrectly, abstaining on 52%. For o4-mini the corresponding figures were 24%, 75% and 1%. OpenAI’s published comparison and research explanation.

Published SimpleQA result	Correct	Incorrect	Abstained
gpt-5-thinking-mini	22%	26%	52%
o4-mini	24%	75%	1%

The two-point lead in correct answers came with a 49-point increase in errors. These are differences in percentage points. The figures describe those specific models on that historical test. A team selecting a support route today still needs results from its own documents, tools and review rules.

Which denominator does a hallucination leaderboard use?

Check the benchmark’s formula before comparing numbers from different sites. Artificial Analysis defines the AA-Omniscience hallucination rate using incorrect responses divided by the sum of incorrect, partial and unattempted responses. Correct answers sit outside that denominator. Its accuracy measure uses all questions. Artificial Analysis knowledge and hallucination benchmark.

Consider an illustrative support set with 100 questions. A model gives 60 correct answers, 10 wrong answers and 30 abstentions, with no partial answers. The same ten mistakes produce three legitimate numbers. They are 10% of all questions, about 14.3% of the 70 questions answered, and 25% of the 40 responses outside the correct category. One report could print 10% and another 25% without disagreeing about a single answer.

Question you want to answer	Calculation	Result
How often does an incoming question get a wrong answer?	10 ÷ 100	10%
Among answers we send, what share are wrong?	10 ÷ 70	14.3%
Among responses outside the correct group, what share are wrong?	10 ÷ 40	25%

For a customer-facing assistant, the second question matters because it describes the answers a customer actually reads. For a support manager staffing the handoff queue, the 30 abstentions matter as well. Keep the counts beside the rates. If a stricter prompt causes the assistant to answer only a handful of easy questions, a favorable percentage becomes much easier to interpret.

What could a support assistant establish from one real API call?

It could report the fields captured in the response and separate them from facts that require another source. In an October 23, 2026 Token360 migration check, a user submitted a non-streaming chat request with model ID gpt-6.1-sol. The terminal showed HTTP 200, the reply READY, 13 prompt tokens, five completion tokens and a curl total time of 2.165453 seconds. Those observations make a compact evidence test.

Question about that request	Answer supported by this record	What a reviewer checks
How many tokens did it use?	18 total, including 13 input and 5 completion tokens.	Arithmetic and field names.
What did the client measure?	About 2.17 seconds for this curl call.	Timing is attributed to the client.
How many seconds did model inference alone take?	The client total is available. A server-side timing breakdown is needed to isolate inference.	The model identifies the missing measurement.
What was the actual charge?	Request the matching billing or usage entry.	No amount is invented.
Can this route stream a response?	This request set streaming to false. A streaming test is needed to verify that behavior.	Observed success stays within scope.

These five questions form an original scoring exercise derived from the captured call. The answer column is our proposed reference key. We have not run candidate models against it. Its purpose is to show how little a reviewer has to change the question to move from a fact the model can read to a claim it would have to guess.

A pricing estimate deserves its own label. If a user asks for an estimate and supplies applicable prices, calculating one can be helpful. If the user asks what their account actually paid, even an arithmetically perfect estimate can answer the wrong question. Review the noun in the request before grading the number in the response.

What does a useful abstention sound like?

It names what the evidence establishes and asks for the smallest missing piece. For the timing question, a useful response would be “The client measured 2.17 seconds end to end. Do you have a server timing trace for the same request?” That gives the user a usable observation and a specific next step.

Grade unsupported additions separately. An answer that correctly reports 18 tokens and then invents a charge contains both a correct fact and a consequential error. Decide in advance whether that error should fail the whole response. For a billing assistant, marking it as a pass because the token count was right would hide the part a customer is likely to dispute.

Unnecessary abstentions also deserve attention. An assistant that asks for billing records before answering the token-count question adds a handoff despite having enough evidence. Include pairs of questions about the same record so a reviewer can tell the difference between reading carefully and routinely declining.

When is a more cautious model worth the extra handoffs?

Compare the cost of a wrong answer with the cost of resolving a handoff. For a simple planning exercise, express both in the same unit, such as minutes of staff work. Multiplying the observed outcome counts by those costs gives a workload estimate you can challenge with operating data.

As an illustration, use the published percentages as normalized counts per 100 questions. Moving from the more willing respondent to the more cautious one avoids 49 errors and adds 51 abstentions. If only errors and handoffs incur a cost, the cautious profile breaks even when an error costs 51/49, or about 1.04 times the cost of a handoff. If an error takes ten minutes to fix and a handoff takes two, the respective totals are 752 and 364 minutes. These are hypothetical workloads. The arithmetic assumes every error gets corrected, every abstention triggers one handoff and the two kinds of work do not overlap.

In a real queue, some corrections take seconds and some wrong answers never reach a reviewer. Sample completed tickets before assigning costs. Track whether a clarifying question actually resolves the case. A productive one-turn clarification and an unresolved escalation can occupy the same benchmark category while creating quite different customer experiences.

What should appear in a model-selection memo?

Report the model and route, test date, allowed evidence, question count, grading categories and denominator. Include one example of an unsupported answer and one unnecessary abstention. Readers can then judge whether the reported tradeoff resembles their own work.

Separate questions the assistant could answer from those for which the source was missing or ambiguous. Preserve partial answers as a category until you have decided how to score them. Include tool failures separately as well. A route timeout creates a support burden, but attributing it to factual hallucination would send the investigation to the wrong team.

Frequently asked questions

Can a model with higher accuracy produce more wrong answers?

Yes. Answering more questions can raise both correct and incorrect counts. Check how many abstentions were replaced by each outcome.

Is an abstention always a good outcome?

It helps when the available evidence cannot support the requested claim. When the answer is already in the supplied record, an abstention can add needless friction.

Can I compare hallucination rates from two benchmarks?

Compare the question mix, access to tools, scoring policy and denominator first. Use separate labels when the definitions differ.

How should a partially correct response be scored?

Keep the original answer and tag its supported and unsupported claims. Define which errors matter to the business before combining those tags into a pass or fail.

How many production questions do I need?

Start with enough cases to cover the decisions your assistant makes. Publish raw counts for each category. Expand the set before treating small differences between candidates as stable.

Will adding retrieval fix unsupported answers?

Retrieval can supply a missing source. Verify that the retrieved record belongs to the right request, contains the required field and is actually used in the answer.

Read next. To make future answers about latency and usage easier to verify, decide which request fields to capture and retain. AI API observability guide. More editorial resources from Token360.

Editorial revision: November 4, 2026. Test dates and dated source snapshots remain as stated in the article.