The Machine Learning Workflow & Lifecycle

ML projects don't end at deployment

A common mistake among teams new to production ML is treating a model like a traditional software feature: build it, ship it, move on. ML systems don't work that way, because the real world the model was trained on keeps changing after launch — user behavior shifts, upstream data sources change schema, competitors change the landscape your fraud or pricing model operates in.

The ML lifecycle is better understood as a loop with seven stages, where the output of monitoring feeds back into problem framing and data collection for the next iteration. Skipping or under-investing in any one stage tends to surface as a production incident months later, usually in a stage downstream of the one that was actually neglected.

1. Problem framing

Before any data is touched, the hardest and most frequently skipped step is translating a business problem into a well-posed ML problem: what exactly is being predicted, what's the unit of prediction (a user, a session, a transaction?), what does success look like in a business metric, and — critically — what's the cost of a false positive versus a false negative. A churn model optimized for accuracy might be useless if the business actually cares about catching the 5% of highest-value accounts about to churn, which a naive accuracy-optimized model will happily misclassify since they're a small fraction of the data.

2. Data collection and labeling

This stage determines the ceiling of everything downstream — no amount of modeling sophistication compensates for data that doesn't represent the problem you're solving. Key decisions: where labels come from (human annotation, implicit signals like clicks, weak/heuristic labeling), how to avoid label leakage (using information that wouldn't be available at prediction time), and how to ensure the collected data matches the distribution the model will actually see in production, not a convenient but biased proxy (e.g., labeling only from users who completed an action, ignoring those who dropped off).

3. Training

This is the stage most engineers think of as 'doing ML' — selecting an architecture, tuning hyperparameters, running the training loop. In practice, well-run teams spend less time here than the surrounding stages, because a clean dataset and a correctly framed problem make training comparatively mechanical, while a messy dataset makes training an endless, low-leverage tuning exercise. See Training ML and Hyperparameter Tuning for the mechanics of this stage specifically.

4. Evaluation

A model is evaluated against held-out data the training process never saw, using metrics chosen during problem framing rather than whatever's convenient (accuracy is frequently the wrong metric for imbalanced problems like fraud or churn — precision/recall, cost-weighted metrics, or calibration are usually more relevant). This stage should also include slice-based evaluation — checking performance separately across important subgroups (by region, by device, by customer tier) because an aggregate metric can look healthy while the model fails badly for a specific, business-critical slice.

5. Deployment

Getting a model artifact into a system that serves real traffic — covered in depth in Model Serving & Inference Systems. The key engineering concerns at this stage are rollout safety (canary releases, shadow traffic, gradual ramp-up) and reversibility: you need to be able to roll back to the previous model version at least as fast as you'd roll back a bad code deploy, which means model versions need to be artifacts you can swap, not retraining jobs you have to rerun under pressure.

6. Monitoring

Once live, the model needs continuous observation of both its inputs (is incoming data drifting from what it was trained on?) and its outputs (is prediction quality degrading?), plus the downstream business metric it was built to move. This is the stage most organizations under-invest in, because unlike training or deployment it produces no visible deliverable — it just quietly prevents the slow, silent decay covered in depth in MLOps: Monitoring & Maintaining Models in Production.

7. Retraining

Monitoring eventually surfaces a signal — drift, degraded accuracy, a changed business requirement — that triggers retraining. Retraining isn't just rerunning the same training script on newer data; it should revisit problem framing (is the original objective still the right one?) and data collection (do we need labels for the new failure modes monitoring surfaced?) before jumping straight back to training. This is exactly why the lifecycle is drawn as a loop rather than a pipeline with an end.

Traditional software release vs. ML lifecycle

What's next

Every stage of this loop depends on the quality of the data feeding it. The next lesson, Data Engineering for Machine Learning, goes deep on the data collection and feature pipelines that make — or quietly break — everything downstream.