A classifier that outputs "95% confident this is fraud" implies something specific: among all the times it says 95%, roughly 95% of those predictions should be correct. Model calibration is the property that a model's predicted confidence scores actually match its empirical accuracy. A well-calibrated model's 70%-confidence predictions are right about 70% of the time; a miscalibrated model's 70%-confidence predictions might be right 50% of the time, or 95% of the time — the number is just a score, disconnected from reality.

Calibration is a distinct property from accuracy. A model can be highly accurate and badly calibrated at the same time — it gets the right answer often, but its stated confidence doesn't mean what it claims to mean. Modern deep neural networks are a well-known example: they tend to be substantially overconfident, assigning very high probability to predictions that are wrong meaningfully more often than that probability implies.

Note

A softmax layer produces numbers that sum to 1 and look like probabilities, but nothing in standard cross-entropy training forces those numbers to be *calibrated* probabilities. A network can learn to push softmax outputs toward 0 and 1 simply because doing so minimizes training loss, independent of whether the resulting confidence reflects true predictive uncertainty.

Calibration curves and reliability diagrams

The standard way to visualize calibration is a reliability diagram (or calibration curve): bucket predictions into confidence bins (e.g., 0-10%, 10-20%, ..., 90-100%), and for each bin, plot the model's average predicted confidence against the actual fraction of correct predictions in that bin. A perfectly calibrated model traces the diagonal line y = x. A curve that sits below the diagonal in the high-confidence region shows overconfidence — the model claims more certainty than its actual accuracy supports.

The single-number summary of this gap is Expected Calibration Error (ECE): the weighted average, across bins, of the absolute difference between average confidence and average accuracy in that bin. Lower is better; zero would mean perfect calibration.

python

Temperature scaling

The most widely used fix for neural network miscalibration is temperature scaling: divide the model's logits (pre-softmax outputs) by a single learned scalar T before applying softmax. T > 1 softens the probability distribution, pulling overconfident predictions back toward less extreme values; T < 1 sharpens it. Because T is a single scalar fit on a held-out validation set after training, it reshapes confidence without touching the model's actual predictions (which class gets the highest score never changes), so accuracy is completely unaffected — only the reported confidence shifts.

T is typically fit by minimizing negative log-likelihood (equivalent to cross-entropy) on a validation set, holding the trained model's weights frozen. It's cheap, requires no retraining of the underlying model, and in practice recovers most of the calibration gap for modern deep networks.

python

Other calibration methods

Platt scaling fits a logistic regression on top of a model's raw scores, mapping them to calibrated probabilities — common for SVMs and other models that don't natively output probabilities. Isotonic regression fits a non-decreasing step function mapping raw scores to calibrated probabilities, more flexible than Platt scaling's fixed sigmoid shape but requiring more validation data to fit reliably without overfitting the calibration mapping itself. All of these methods share the same contract: they're fit post-hoc on a held-out validation set, separate from training data, and they adjust confidence without changing which class is predicted.

Calibration methods compared

Method	Changes predicted class?	Flexibility	Typical use case
Temperature scaling	No	Low — one global scalar	Neural networks, especially multi-class
Platt scaling	No	Medium — fixed sigmoid shape	SVMs and other non-probabilistic scorers
Isotonic regression	No	High — arbitrary monotonic mapping	Larger validation sets, binary classification

Note

If a model's output only feeds a top-1 decision (just take the highest-scoring class), calibration barely matters — accuracy is what counts. But when confidence feeds a downstream decision threshold, a risk score, a human triage priority, or anywhere probabilities get combined or compared across examples (medical diagnosis, credit risk, autonomous systems), miscalibration can cause systematically wrong decisions even from an accurate model.

What's next

Calibration adjusts the confidence of a single trained model. Another way to get more reliable, better-calibrated-feeling predictions is to combine multiple models' outputs. The next lesson, Snapshot & Neural Network Ensembles, covers how to build an ensemble cheaply from a single training run instead of training N separate models from scratch.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →