Hallucination Rate Is the Wrong Metric for Most Production Use Cases

A stakeholder once asked me, with total sincerity, 'what's our hallucination rate.' I understood why they wanted a number — it's clean, comparable, reportable to leadership in one slide. But I couldn't give them one that meant anything, because hallucination rate as a single aggregate number flattens a dozen genuinely different failure modes that matter in completely different ways to the actual product.

Not all hallucinations cost the same

A model inventing a plausible-sounding but wrong product SKU in a support chatbot is a different severity of problem than a model slightly over-summarizing a document's tone in a low-stakes internal digest. Both might get counted identically in an aggregate 'hallucination rate,' and that single number tells a stakeholder nothing about which one to worry about or where to spend engineering effort fixing it.

A 3% hallucination rate on a legal summary tool and a 3% hallucination rate on a meme caption generator are not the same risk, no matter how identical the number looks on a slide.

What we measure instead

• Factual error rate on claims that are independently verifiable against a source document — the clearest, narrowest definition of hallucination worth tracking in isolation.
• Unsupported-claim rate: statements the model makes that aren't in the provided context at all, separate from claims that are simply wrong.
• Severity-weighted error rate, where an error in a high-stakes field (a price, a medical claim, a legal statement) is weighted far above a stylistic overreach.
• Downstream impact rate — of the errors that occur, how many actually change a user's decision or action, versus how many are cosmetically wrong but inconsequential.

Report the breakdown, not the average

When that stakeholder asked again a month later, I gave them a table instead of a number: error rate by field, weighted by what happens downstream if that specific field is wrong. It took longer to explain than 'our hallucination rate is 4%,' but it's the only version of that answer that actually tells anyone whether to be worried, and about what.