It's tempting to run two models, compare their accuracy on a test set, and declare the higher number the winner. But a test set score is a single noisy estimate. If model A scores 84.2% and model B scores 83.9% on the same test set, is that a real improvement, or could the difference have happened by chance from how the test set happened to be split and sampled?

Statistical hypothesis testing gives you a principled way to answer that question instead of eyeballing it. The core idea: assume there is no real difference between the two models (the null hypothesis), then ask how likely it would be to see a gap this large — or larger — purely by chance under that assumption. If that probability (the p-value) is very small, you reject the null hypothesis and conclude the difference is likely real.

Why a single test-set comparison is weak evidence

A single held-out test set gives you one sample of performance difference. That one number is subject to sampling variability — a different random test set drawn from the same distribution could easily shift the ranking. The fix mirrors what cross-validation already does for evaluating a single model: instead of one comparison, generate multiple paired comparisons (e.g., one score per cross-validation fold for each model) and treat the resulting set of differences as a distribution you can test statistically, rather than a single anecdote.

The paired t-test for comparing model scores

When you have two models evaluated on the same folds or the same resampled datasets, use a paired t-test. Pairing matters because it controls for the fact that some folds are simply harder than others for both models — you're testing whether, fold by fold, one model consistently does better, not just comparing two unrelated distributions of scores.

For each fold $i$, compute the difference $d_i = \text{score}_A^{(i)} - \text{score}_B^{(i)}$. The paired t-test statistic is:

$$t = \frac{\bar{d}}{s_d / \sqrt{n}}$$

where $\bar{d}$ is the mean of the differences, $s_d$ is their standard deviation, and $n$ is the number of folds. A large $|t|$ (combined with a small p-value) indicates the average difference is unlikely to be zero — model A's edge over model B is probably not noise.

python

Note

A technical wrinkle: the standard paired t-test assumes independent samples, but cross-validation folds overlap in training data (each fold's model is trained on almost the same data as the others), which tends to underestimate the true variance and inflate significance. This is a well-known limitation. In practice, many practitioners still use the paired t-test as a useful heuristic, but more rigorous alternatives exist — such as the corrected resampled t-test, or simply repeating the entire cross-validation procedure with different random seeds and comparing the resulting distributions.

McNemar's test for comparing classifiers on one test set

When you only have a single test set (not multiple folds) and are comparing two classifiers' predictions on the exact same examples, McNemar's test is the standard tool. It builds a 2x2 contingency table of how often each model got an example right or wrong, focusing specifically on the examples where the two models disagreed:

	Model B correct	Model B incorrect
Model A correct	both right	A right, B wrong
Model A incorrect	A wrong, B right	both wrong

The test statistic looks only at the two "disagreement" cells (A right/B wrong, and A wrong/B right) and asks whether that disagreement is lopsided enough to be unlikely under the null hypothesis that the two models are equally accurate. It is well-suited to the common situation where retraining many cross-validation folds is expensive, but you already have both models' predictions on one shared test set.

Choosing a significance test for model comparison

Situation	Recommended test	What it compares
Multiple CV folds, same splits, continuous metric	Paired t-test	Mean of per-fold score differences
Single shared test set, classification	McNemar's test	Disagreement pattern between the two models' predictions
Non-normal score distributions, small sample	Wilcoxon signed-rank test	Median of paired differences (nonparametric)
Comparing more than two models at once	Friedman test + post-hoc	Ranks across all models simultaneously

Note

With a large enough test set, even a trivial 0.1% accuracy difference can be statistically significant — the p-value answers "is this difference real?" not "is this difference big enough to matter?" Always pair a significance test with a practical judgment: does the gap justify the cost of switching models, added latency, or reduced interpretability? A statistically significant but practically tiny improvement is rarely worth shipping.

What's next

Hypothesis testing gives you rigor when comparing models you've already built. The next lessons shift focus to the models themselves — starting with activation functions, one of the most consequential and frequently misunderstood design choices inside a neural network.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →