Bootstrapping & Resampling Methods

The problem: you only have one sample

You trained a model and measured 87% accuracy on your test set. Is that a reliable number, or would a different test set have given you 82%? Classical statistics answers this with formulas — if you know your statistic follows a known distribution, you can derive its standard error analytically. But most statistics you care about in practice (a model's F1 score, the median of a skewed distribution, the difference between two correlation coefficients) don't have a clean formula for their uncertainty.

Bootstrapping sidesteps the problem entirely. Instead of deriving uncertainty mathematically, you simulate it: you treat your one sample as a stand-in for the population, and you resample from it repeatedly to see how much your statistic varies.

The core idea: resampling with replacement

Given a dataset of n observations, a bootstrap sample is created by drawing n observations from it with replacement — meaning the same original observation can appear multiple times, and others may not appear at all. Each bootstrap sample is the same size as the original, but it's a slightly different mix.

You compute your statistic of interest (mean, accuracy, median, whatever) on each bootstrap sample. Do this thousands of times, and you get a distribution of that statistic — the bootstrap distribution. Its spread tells you how much your statistic would plausibly vary if you had collected a different sample from the same underlying population.

The percentile method vs. other CI constructions

The code above uses the percentile method: run n_boot resamples, sort the resulting statistics, and take the 2.5th and 97.5th percentiles as a 95% interval. It's simple and works well in most cases, but it can be biased when the bootstrap distribution is itself skewed.

Two refinements are common in production statistics libraries: the bias-corrected and accelerated (BCa) method adjusts the percentile cutoffs to correct for skew and bias in the bootstrap distribution, and the studentized bootstrap rescales each bootstrap statistic by its own estimated standard error before computing percentiles. Both are more accurate but more expensive to compute. For most ML use cases — comparing model metrics, sanity-checking a reported number — the plain percentile method is sufficient.

Bootstrapping model performance metrics

The most common ML use case is putting a confidence interval around a test-set metric. Instead of resampling raw data points and recomputing a mean, you resample the predictions and labels together (keeping each prediction paired with its true label) and recompute accuracy, F1, AUC, or any other metric on each resample.

This matters because a single accuracy number from a 1,000-row test set can swing several percentage points just from which rows happened to land in the test set. A bootstrap CI tells you whether a 2-point improvement between two models is a real signal or just noise in a small test set.

Resampling methods compared

Where this shows up beyond confidence intervals

Bootstrapping is also the statistical engine behind bagging (bootstrap aggregating) — the technique underlying random forests. Each tree in a random forest is trained on a separate bootstrap sample of the training data, and averaging their predictions reduces variance in exactly the way averaging bootstrap statistics reduces uncertainty in an estimate. Understanding bootstrapping as a confidence-interval tool also explains why bagging works: training on resampled data isn't a data augmentation trick, it's the same resampling logic applied to model training itself.

What's next

Bootstrapping gives you a confidence interval without committing to a probability model, which is the empirical, 'let the data speak' approach to uncertainty. The next lesson, Bayesian vs. Frequentist Statistics, puts that empirical approach in context against its philosophical alternative — using probability to represent belief rather than just long-run frequency — and shows where each shows up in day-to-day ML practice.