A/B Testing an ML Model in Production Taught Us More Than Offline Eval

Offline, the new ranking model beat the incumbent on every metric we tracked — higher NDCG, better precision at the top of the ranking, cleaner on every holdout split we threw at it. We were ready to ship it as a straight replacement. Production A/B testing instead of a direct swap is what stopped us from shipping something that was actually worse for a meaningful chunk of real users.

What the offline numbers hid

The aggregate production metrics in the A/B test matched the offline story at first glance — overall engagement up, overall conversion up. But when we sliced the A/B results by user segment, a pattern showed up that no offline holdout had surfaced: the new model was a clear win for high-activity users and a measurable regression for new, low-activity users, a segment that was underrepresented enough in our offline training and eval data that its degradation got averaged away.

An aggregate metric going up in an A/B test can hide a segment going down. Offline eval almost never has enough of the underrepresented segment to catch it at all.

Why offline eval missed it and A/B testing caught it

Offline eval ran on a static holdout that mirrored our training distribution, which meant the new-user segment was exactly as underrepresented in the eval set as it was in training — the exact condition that lets a regression slip through unnoticed. Production A/B testing exposed the model to the real, live distribution of traffic, including the full volume of new users the offline set had quietly underweighted, and the segment-level dashboard we had running on the test is what actually surfaced the gap.

• Offline holdouts inherit the same representation gaps as training data — they can't catch what they don't have enough of.
• Segment-level A/B analysis, not just the aggregate top-line metric, is where regressions on minority segments actually show up.
• A model can be a net win and a targeted loss at the same time — ship decisions need to account for both.
• Staged rollout, not full replacement, gave us room to catch this before it hit 100% of traffic.

We shipped a segmented solution in the end — the new model for high-activity users, the old model retained for new users until we had enough data to properly retrain for that segment. Offline eval told us the model was better on average. Only the production test told us better on average wasn't the same as better for everyone, and for a model making ranking decisions that shape what new users see first, that distinction mattered.