A Confidence Interval Got Misread by a Stakeholder, Here's How We Fixed Communication

We shipped a results readout showing a feature's lift at +3.2%, 95% CI [-0.4%, 6.8%]. A VP looked at the lower bound being negative and told the team to kill the feature because 'there's a real chance it's actually hurting us.' The feature was, by every other signal we had, working. The confidence interval wasn't the problem. How it got read was.

What actually went wrong

A 95% confidence interval doesn't mean there's a 95% chance the true value sits inside it on any given reading — it means that if you repeated this exact experiment many times, 95% of the intervals you'd construct would contain the true effect. That's a statement about the procedure, not a probability statement about this specific interval. The VP read the lower bound crossing zero as 'there's meaningful odds this is negative,' when the actual takeaway was closer to 'we don't yet have enough precision to rule out zero, but our point estimate and every other metric agree it's positive.'

The confidence interval was correct. The sentence describing it to a non-statistician was the actual bug.

The fix wasn't teaching stats to executives

I tried the 'let me explain confidence intervals' route once, in the room, and watched eyes glaze over in real time. That's not a scalable fix and it's disrespectful of people's time — the fix had to live in how we presented the number, not in upskilling every stakeholder on frequentist inference. We changed the readout format permanently.

• Lead with the point estimate and a plain-language confidence statement: 'we're fairly confident this is positive, with a small chance it's closer to flat' instead of raw interval notation.
• Pair every interval with a probability-of-positive-effect number from a quick Bayesian reframe, since that maps directly onto the question people actually want answered.
• Never show a bare interval without a one-line interpretation sitting right next to it — if the chart needs a stats background to read correctly, it's the wrong chart for this audience.
• Flag borderline results explicitly as 'not yet conclusive, recommend extending' rather than letting the raw numbers imply a verdict they don't support.

We re-ran the same result through the new format a week later with more data: +3.4%, and this time the summary line read 'highly likely positive, recommend keeping.' Same underlying statistics, completely different stakeholder reaction. The feature stayed. The lesson wasn't about rigor — the rigor was already there. It was that a correct number presented in the wrong format is functionally the same as a wrong number.