Shadow Deployment Caught a Regression Offline Eval Completely Missed

A new recommendation model cleared every offline accuracy check with room to spare, and the plan was to A/B test it against a slice of live traffic next. Before doing that, we shadow-deployed it — running it in parallel on real production requests, logging its outputs and latency without actually serving them to users. That shadow run is the only reason we caught a latency regression serious enough to have degraded the product if we'd gone straight to an A/B test.

What offline eval couldn't see

Offline evaluation measures a model in isolation, usually batched for efficiency, with no sense of the request-level latency it'll produce under real serving conditions. The new model used a feature that required a real-time lookup against a slower backing store than the features the old model relied on. In batch offline eval, that lookup got amortized across thousands of requests and barely registered. Under live, per-request serving conditions, it added close to 400ms to the median response time.

Offline accuracy tells you whether a model is right. Shadow deployment is what tells you whether it's fast enough to matter in production at all.

Why shadow deployment specifically, not just a canary

We could have found this with a canary release too, eventually — but a canary exposes a small slice of real users to the regression while you're figuring out what's wrong. Shadow deployment let us run the full production request volume through the new model's code path with zero user-facing risk, because the shadow output never got served. We caught the latency spike from internal logging alone, root-caused it to the new feature's lookup pattern, and fixed it with a caching layer before a single real user ever touched the slow path.

• Shadow deployment exposes real-world latency and resource usage that batch offline eval can't.
• Zero user-facing risk during the shadow period — regressions get caught without anyone experiencing them.
• Requires the infrastructure to run two model paths per request without doubling response time for the live path — worth building once, reused on every model change after.
• Doesn't replace A/B testing for measuring actual user-facing impact, but it should run before an A/B test, not instead of one.

The accuracy win on the new model was real and we eventually shipped it. But if shadow deployment hadn't been standard practice on this team already, the first anyone would have heard about the latency problem is a spike in page load complaints during the A/B test, with real users on the losing end of the regression the whole time.