A Model Registry Saved Us During an Emergency Rollback

A newly deployed pricing model started producing scores that were visibly off within an hour of launch — not subtly wrong, obviously wrong, the kind of thing a stakeholder notices immediately and escalates within minutes. The incident itself was stressful. The rollback wasn't, and that's entirely because of a model registry we'd set up months earlier mostly out of habit.

What the registry actually gave us

Every model version that had ever been deployed to production was logged in the registry with its exact artifact, the training data snapshot it came from, its validation metrics, and the timestamp it went live. When the pricing scores went sideways, the question wasn't "what was running before this" followed by a scramble through deploy logs and Slack history — it was a lookup that took about thirty seconds to answer definitively.

A rollback is only as fast as your ability to answer 'what exactly was running before' with certainty instead of a best guess.

How the actual rollback went

We pulled the previous version's exact artifact from the registry, confirmed it was the one validated and running stably for the two weeks before the bad deploy, and pointed the serving layer back at it. Four minutes from "something's wrong" to "old model is live again." No guessing which of three possible prior versions was the right one, no risk of rolling back to a version that had its own known issue from before.

Incident detected → registry lookup: last known-good version + metadata
→ confirm artifact hash matches what was actually running
→ redeploy exact artifact → rollback complete

Total time: ~4 minutes

The root cause turned out to be a data pipeline issue upstream of the model feeding it stale features, not the model itself — a separate investigation that took the rest of the day. But the business impact of the incident was contained to under five minutes specifically because rolling back wasn't a research problem. A model registry is unglamorous infrastructure right up until the day you need to answer "what was running" with total confidence, fast.