Confidence Intervals for ML Metrics
A single number hides how much you actually know
'Our model gets 91% accuracy' sounds precise, but it's missing a crucial piece of information: how much would that number move if you evaluated on a different test set drawn from the same distribution? On a 10,000-row test set, 91% is probably stable to within a point. On a 100-row test set, it could easily have been 85% or 95% with a different random split.
A confidence interval answers this directly: instead of reporting a point estimate, you report a range that's likely to contain the true metric, along with a stated confidence level (typically 95%). Every metric you report from a finite test set carries sampling uncertainty, and reporting that uncertainty is what separates a rigorous evaluation from a number pulled out of one lucky (or unlucky) run.
Note
The analytic approach: Wilson score interval for accuracy
Accuracy is a proportion (correct predictions / total predictions), so it follows a binomial distribution, and there's a closed-form way to compute its confidence interval. The naive approach — the normal approximation (accuracy ± 1.96 × standard error) — breaks down near 0% or 100% accuracy, where it can produce intervals outside [0, 1]. The Wilson score interval corrects for this and is the standard choice for proportions in practice.
The general-purpose approach: bootstrap for anything
The Wilson interval works for accuracy specifically because accuracy is a simple proportion. But precision, recall, F1, AUC, and most other metrics don't have a clean closed-form interval — their sampling distributions are more complicated or depend on correlations between errors. For these, the bootstrap is the standard tool: resample (prediction, label) pairs with replacement, recompute the metric on each resample, and take percentiles of the resulting distribution.
The tradeoff is computational cost versus generality. The Wilson interval is instant and exact for its one use case; the bootstrap is slower but works identically for any metric you can compute, which is why it's the default recommendation for anything beyond plain accuracy.
Choosing a CI method by metric
| Metric | Recommended method | Why |
|---|---|---|
| Accuracy | Wilson score interval | Closed-form, exact for binomial proportions, stable at extremes |
| Precision / Recall | Bootstrap | Still proportions, but conditioned on predicted/actual positives — no simple closed form |
| F1 score | Bootstrap | Harmonic mean of two proportions, no closed-form sampling distribution |
| AUC-ROC | Bootstrap, or DeLong's method | DeLong's test is exact but more complex to implement; bootstrap is simpler and general |
| RMSE / MAE (regression) | Bootstrap, or normal approximation with large n | Normal approximation is fine when n is large and errors aren't heavily skewed |
Note
What's next
Confidence intervals tell you how much uncertainty exists in the data you already collected. But they can't tell you whether you collected *enough* data to detect a difference you actually care about before you started. The next lesson, Power Analysis & Sample Size Calculation, covers how to work that out in advance.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.