A/B Testing ML Models in Production

Why offline metrics aren't enough

A model that wins on held-out test accuracy doesn't automatically win in production. Offline evaluation measures the model against a frozen, historical dataset, but production traffic is live, drifting, and shaped by feedback loops the offline set can't capture — a recommendation model's offline precision says nothing about whether users actually click more, and a fraud model's offline AUC says nothing about whether fraudsters adapt once the new model is deployed. The only way to know whether a model variant genuinely improves the metric you care about is to expose it to real traffic and measure the real outcome.

This is why mature ML teams treat online A/B testing as a mandatory gate before full rollout, not an optional nicety. Offline metrics decide which candidates are worth testing; online metrics decide which candidate ships.

Designing a model A/B test

A model A/B test randomly splits incoming traffic (or users, sessions, or requests, depending on the unit of randomization) between a control arm served by the current production model and one or more treatment arms served by challenger models. Three design choices matter more for models than for typical UI experiments: the randomization unit must be stable enough that the same entity doesn't flip between arms mid-session (user-level randomization is usually safer than request-level for anything with session continuity); the primary metric must be a true business or user outcome — click-through, conversion, fraud dollars caught, latency-adjusted engagement — not a proxy like model confidence; and the guardrail metrics must catch harm the primary metric wouldn't show, such as latency regressions, error rates, or fairness metrics across subgroups.

Sample size and test duration should be computed in advance using the expected effect size and the metric's natural variance, not decided by eyeballing a dashboard until it looks good — this is the single most common source of bad experiment decisions.

Offline evaluation vs. online A/B testing

Statistical pitfalls specific to model tests

Model A/B tests fail in a few recurring ways that generic experimentation platforms don't always protect against. Peeking — checking results daily and stopping as soon as significance is hit — inflates the false positive rate dramatically, because you're effectively running many sequential tests and taking the first lucky one; use a pre-registered sample size or a proper sequential testing method instead. Network effects and interference occur when one arm's predictions affect the other arm's outcomes — a pricing model in treatment that lowers prices can shift demand patterns that control also experiences through shared inventory, violating the independence assumption underlying standard significance tests. Novelty and primacy effects mean a new model's initial lift (or dip) can fade or grow over the first days of exposure as users adapt, so short tests on models with behavioral feedback loops can systematically mislead. Heterogeneous treatment effects — a model that helps power users but hurts new users — can cancel out to a null aggregate result that hides a real, actionable finding; always slice results by meaningful segments, not just the topline average.

Champion-challenger as a continuous process

Rather than a one-off test, many production ML systems run a standing champion-challenger framework: the current production model is the champion, and every new candidate must beat it on a live traffic slice before being promoted. This turns model improvement into a continuous, measurable pipeline rather than a series of risky big-bang deployments, and it naturally produces an audit trail of why each model replaced its predecessor — valuable both for debugging regressions and for regulatory review in domains like credit and insurance.

What's next

A/B testing tells you whether a new model is better once it's already receiving real traffic — but you still need a safe way to get it there. Continue to Shadow Deployment & Canary Releases for ML to see how teams validate a challenger model's behavior before it ever risks a real user outcome.