Model Version Pinning Saved Us From a Silent Regression

After getting burned by a silent model-update regression on one client's pipeline, I started pinning exact model versions across every production system I touch. It felt a little paranoid until it directly paid off on a different client's project six months later.

How it actually played out

Staging was configured to track the provider's 'latest' alias, production was pinned to a specific dated version. A provider-side update rolled out to the 'latest' alias and staging's classification accuracy on a tagging task dropped by about 15% overnight — caught within a day because our eval suite ran against staging automatically. Production, still on the pinned version, was completely unaffected and kept serving the known-good behavior.

Pinning a model version doesn't prevent regressions, it just makes sure they happen somewhere you're watching instead of somewhere your customers are.

The actual workflow

Staging:    tracks "latest" alias  → eval suite runs nightly
Production: pinned to dated version → only promoted after
            staging eval passes AND manual diff review

If staging eval regresses: investigate, do NOT promote.
If staging eval passes: bump production's pin explicitly.

• Pinning costs you the ability to get silent free upgrades, which sounds like a downside until you've been burned by a silent free downgrade instead.
• The eval suite is the part that actually does the work — pinning alone just buys you time to notice a regression before it reaches production.
• Budget for the fact that every provider's pricing and support incentives nudge you toward tracking 'latest' — pinning is a deliberate swim against that current, not the default.

Nobody thanks you for the regression that didn't happen. But the alternative — explaining to a client why their production classifier got worse on a day you changed nothing — is a conversation worth avoiding with a few lines of config.