Calibration Curves and Reliability Diagrams

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Accuracy isn't the whole story

A classifier can be highly accurate yet badly miscalibrated. Accuracy only asks whether the predicted class matches the true class; it says nothing about whether a predicted probability of 0.9 actually corresponds to the event happening 90% of the time. A model that outputs 0.99 for everything it predicts positive, even when it's only right 70% of the time, can still look accurate on a balanced test set while being dangerously overconfident.

Calibration matters whenever the probability itself is used downstream — for risk scoring, triage thresholds, expected-value calculations, or anywhere a human or system treats "0.8" as meaning roughly an 80% chance.

Defining calibration formally

A model is perfectly calibrated if, among all examples assigned predicted probability $p$, the true fraction of positives is also $p$:

$P(y = 1 \mid \hat{p} = p) = p \quad \text{for all } p \in [0, 1]$

In practice no model achieves this exactly at every value of $p$, so calibration is assessed empirically by binning predictions and comparing predicted versus observed frequencies.

Building a reliability diagram

A reliability diagram (also called a calibration curve) is built like this:

1. Take the model's predicted probabilities for the positive class on a held-out set.
2. Sort examples into bins by predicted probability (e.g., [0, 0.1), [0.1, 0.2), ..., [0.9, 1.0]).
3. Within each bin, compute the mean predicted probability and the actual fraction of positives.
4. Plot mean predicted probability (x-axis) against observed fraction of positives (y-axis).

A perfectly calibrated model traces the diagonal line $y = x$. Points above the diagonal mean the model is underconfident in that bin (true rate exceeds predicted); points below mean it's overconfident. Most modern deep networks trained with cross-entropy and heavy regularization or large capacity tend to be systematically overconfident — their curve sags below the diagonal at high predicted probabilities.

Reliability diagram with sklearn

python

import numpy as np
from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

# y_true: ground-truth labels (0/1), y_prob: predicted P(y=1)
prob_true, prob_pred = calibration_curve(y_true, y_prob, n_bins=10, strategy="uniform")

plt.plot([0, 1], [0, 1], linestyle="--", label="Perfectly calibrated")
plt.plot(prob_pred, prob_true, marker="o", label="Model")
plt.xlabel("Mean predicted probability")
plt.ylabel("Observed fraction of positives")
plt.legend()
plt.title("Reliability diagram")
plt.show()

Expected Calibration Error (ECE)

Reliability diagrams are visual; Expected Calibration Error summarizes miscalibration as a single scalar. Partition predictions into $M$ bins $B_1, \ldots, B_M$ by predicted probability. For each bin, let $\text{acc}(B_m)$ be the observed positive rate and $\text{conf}(B_m)$ be the mean predicted probability. Then:

$\text{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} \left| \text{acc}(B_m) - \text{conf}(B_m) \right|$

ECE is a weighted average of the gap between confidence and accuracy across bins, weighted by how many examples fall in each bin. Lower is better; 0 is perfect calibration. A related metric, Maximum Calibration Error (MCE), takes the worst bin instead of the average — useful when you care about the single most dangerous miscalibrated region rather than the overall average.

Computing ECE from scratch

python

import numpy as np

def expected_calibration_error(y_true, y_prob, n_bins=10):
    bin_edges = np.linspace(0, 1, n_bins + 1)
    bin_idx = np.digitize(y_prob, bin_edges[1:-1])
    ece = 0.0
    n = len(y_true)
    for b in range(n_bins):
        mask = bin_idx == b
        if mask.sum() == 0:
            continue
        acc = y_true[mask].mean()
        conf = y_prob[mask].mean()
        ece += (mask.sum() / n) * abs(acc - conf)
    return ece

print("ECE:", expected_calibration_error(np.array(y_true), np.array(y_prob)))

Fixing miscalibration: Platt scaling and isotonic regression

Two standard post-hoc recalibration methods reshape raw scores into calibrated probabilities without retraining the base model:

Platt scaling fits a one-dimensional logistic regression on top of the model's raw scores $z$: $P(y=1 \mid z) = \sigma(Az + B)$, learning scalars $A$ and $B$ on a held-out calibration set. It assumes a sigmoid-shaped miscalibration pattern and works well for models like SVMs or boosted trees whose scores are monotonic but not probability-like.

Isotonic regression fits a non-decreasing step function mapping raw scores to calibrated probabilities, with no parametric assumption about shape. It's more flexible than Platt scaling but needs more calibration data to avoid overfitting, since it has many more effective degrees of freedom.

Both methods require a dedicated calibration split, separate from training and test data — fitting them on training data would just relearn the model's own biases.

Calibration is dataset- and shift-sensitive

A model calibrated on one population can become miscalibrated under distribution shift — e.g., a medical risk model calibrated on one hospital's patient mix may overstate or understate risk at another hospital with a different base rate. Recalibration (even a simple Platt scaling refit) on data from the deployment distribution is often necessary, and calibration should be re-checked periodically in production, not just validated once before launch.

What's next?

Calibration tells you whether predicted probabilities are trustworthy, but it doesn't by itself tell you where to draw the line between predicting positive and negative — that's a separate decision shaped by the costs of different error types and the operating point you choose along the model's score distribution.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →