MLOps: Monitoring & Maintaining Models in Production

Models don't fail loudly — they decay quietly

A deployed model keeps producing predictions indefinitely, with the same confidence and the same API contract, whether or not those predictions are still accurate for the world it's currently operating in. Unlike a server crash or a failed API call, a decaying model generates no errors, no exceptions, no alerts — just a slow erosion of accuracy that often goes unnoticed until someone traces a business metric decline back to it weeks later.

This is why MLOps monitoring is treated as a distinct discipline from standard application monitoring (uptime, latency, error rates). You need all of that, plus a second layer that watches whether the statistical relationship between inputs and the real world the model is predicting has shifted.

Data drift: the inputs have changed

Data drift (or covariate shift) is when the distribution of input features the model receives in production diverges from the distribution it was trained on — the real-world relationship between features and the target may be unchanged, but the model is now being asked to generalize to inputs it has effectively never seen. Causes include seasonal shifts in user behavior, a new customer segment entering after a market expansion, an upstream data source changing its schema or units, or a new UI flow changing what gets logged.

Data drift is detectable without any new labels — you're comparing the statistical distribution of incoming features against the training distribution, using tests like the Kolmogorov-Smirnov test for continuous features, population stability index (PSI), or simple distributional distance metrics like KL or Jensen-Shannon divergence. This is a major practical advantage: you can detect data drift in real time, long before you'd have ground-truth labels to confirm the model is actually wrong.

Concept drift: the relationship itself has changed

Concept drift is when the underlying relationship between inputs and the target changes — the same input now implies a different outcome than it did when the model was trained. A fraud model trained before a new fraud pattern emerged will see transactions that look statistically identical to past legitimate ones but are now actually fraudulent; a pricing model trained before a competitor entered the market will see demand behave differently at the same price points.

Concept drift is fundamentally harder to detect than data drift because the inputs can look completely normal — nothing about the distribution of features needs to shift at all. The only reliable detection mechanism is comparing model predictions against ground-truth outcomes once they become available, which means concept drift detection is often inherently delayed by however long it takes labels to arrive (hours for fraud, weeks for churn, months for loan default).

Data drift vs. concept drift

Building a monitoring dashboard for ML

A production ML monitoring setup typically tracks four layers simultaneously: system health (latency, error rate, throughput — standard application monitoring), input monitoring (feature distributions over time, missing-value rates, schema violations — catches data drift and upstream pipeline breakage), prediction monitoring (the distribution of the model's own outputs — a sudden shift in predicted-class balance or confidence scores is often the earliest visible symptom of drift, even before labels confirm it), and outcome monitoring (model accuracy, precision/recall, or calibration against ground truth, once labels arrive — catches concept drift and confirms whether drift actually hurt accuracy).

The prediction-monitoring layer deserves special attention because it's the cheapest early-warning signal available: you don't need any ground truth to notice that a model which used to predict 'fraud' on 2% of transactions is suddenly predicting it on 8%, and that alone is worth investigating immediately.

Retraining triggers and cadence

Teams generally use one of three retraining strategies, often in combination: scheduled retraining (weekly or monthly, regardless of measured drift — simple, predictable, but can retrain unnecessarily or too slowly for fast-moving domains), threshold-triggered retraining (automatically retrain when a drift metric or an accuracy drop crosses a defined threshold — more responsive, but requires reliable monitoring and can thrash if thresholds are too sensitive), and performance-triggered retraining, which waits specifically for confirmed outcome degradation rather than proxy drift signals — the most conservative approach, appropriate when retraining is expensive or risky and false alarms from drift alone aren't worth acting on.

In practice, fast-moving domains like ad bidding or fraud often use threshold-triggered retraining measured in hours or days, while slower-moving domains like credit risk or demand forecasting are comfortable with scheduled retraining on a monthly or quarterly cadence, because the underlying relationships genuinely change more slowly.

What's next

Monitoring and retraining close the loop described back in The Machine Learning Workflow & Lifecycle — a production model is never really finished, only ever current. From here, deepen the evaluation side of this loop with LLM Evaluation, or revisit Fine-Tuning for how a retraining cycle is actually executed on a pre-trained foundation model.