A model that improves offline accuracy by 2% can leave real-world outcomes completely unchanged — or make them worse. Offline metrics (accuracy, AUC, RMSE) measure how well a model fits a held-out dataset that was collected under old conditions, with an old model (or no model) already shaping user behavior. They say nothing about how users will actually respond when the new model is live and shaping what they see.

This gap is why online evaluation — testing the model change on real traffic and measuring real user-facing outcomes — is the only way to know whether a model change actually helps. A/B testing is the standard tool for that.

The basic mechanics

An A/B test randomly splits users (or sessions, or requests) into two groups: control, who see the existing model, and treatment, who see the new model. Because the split is random, any systematic difference in outcomes between groups can be attributed to the model change itself, not to pre-existing differences between the users in each group.

The randomization unit matters. Randomizing by user (not by request) is usually correct for products where consistency matters — you don't want the same person flipping between old and new behavior request by request, since that both confuses the user and contaminates the measurement with within-user variance.

python

Note

It's tempting to look at a dashboard full of metrics after launching a test and report whichever one moved favorably. This is p-hacking, and it invalidates the statistical guarantees of the test. Pre-register a single primary metric (and a small set of guardrail metrics) before the experiment starts, and commit to acting on that metric regardless of what else shifts.

Choosing the right metric

The primary metric should be as close as possible to the actual business or user outcome you care about, not a proxy for it. For a recommendation model, click-through rate is a weak proxy (clickbait inflates it); session-level engagement, return visits, or downstream conversion are usually better. For a fraud model, the primary metric probably isn't "flagged transactions" but realized fraud losses balanced against false-positive friction for legitimate users.

Alongside the primary metric, track guardrail metrics — latency, error rates, revenue, user complaints — that should not get worse even if the primary metric improves. A model that boosts engagement 5% but doubles page load time is not a win.

Statistical significance and sample size

Before running the test, estimate the sample size needed to detect the smallest effect size you care about, given the metric's natural variance. Underpowered tests are one of the most common failure modes — they run for too short a time or with too few users, then either show no significant effect (when a real effect exists) or get stopped early on a noisy, temporarily significant result.

A common mistake is "peeking" — checking significance repeatedly as data accumulates and stopping as soon as p < 0.05 appears. This inflates the false-positive rate dramatically. Use a pre-registered sample size and duration, or a sequential testing method explicitly designed to allow safe early stopping.

python

Note

Users sometimes react to a change simply because it's different, not because it's better — a novelty effect that fades after days or weeks. Conversely, users accustomed to the old behavior may resist a genuinely better model at first (a primacy effect), making it look worse than it is. Running tests long enough to let these transients settle, and checking whether effects persist in later days of the test, protects against both false positives and false negatives.

Offline evaluation vs. online A/B testing

Dimension	Offline evaluation	Online A/B test
What it measures	Fit to a static, historical dataset	Real user behavior under the new model
Speed	Fast — minutes to hours	Slow — days to weeks for statistical power
Cost of being wrong	Low — no real users affected	Real — exposes some users to a worse experience
Captures feedback loops	No	Yes — including effects the model itself causes
Typical role	Filter candidate models before testing	Final validation before full rollout

What's next

A/B testing validates whether a model change helps on average — but some algorithms are built specifically to find the unusual cases hiding within normal-looking data. The next lesson turns to anomaly detection, starting with a deep dive into Isolation Forest.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →