ML System Design Fundamentals

Code is no longer the whole system

In traditional software, behavior is fully determined by the code you write. Given the same inputs, a correctly functioning program produces the same outputs, and a bug is something you can usually find by reading the code.

In an ML system, behavior is determined by code plus data plus a trained model artifact — and the model artifact itself is the product of a training process that is hard to fully inspect or reason about line by line. This is the single biggest conceptual shift for engineers moving from traditional software to ML systems: your system's behavior lives partly in a few thousand lines of code, and partly in a few hundred million (or billion) learned weights that no one wrote by hand and no one can fully explain.

Non-determinism is a feature, not a bug — until it isn't

Many ML systems are explicitly non-deterministic: the same input can produce slightly different outputs due to randomness in sampling (temperature-based generation in LLMs), floating-point non-associativity across hardware, or model ensembling. This is often intentional — you want a chatbot to sound natural rather than robotically identical every time.

But non-determinism becomes an engineering problem the moment you need reproducibility: debugging a production incident, running regression tests, or proving to an auditor that a model behaved consistently. Production ML systems typically need a deterministic mode (fixed seeds, temperature zero, pinned hardware/library versions) available specifically for testing and incident replay, even if the default user-facing behavior stays stochastic.

Train/serve skew: the classic silent killer

Train/serve skew (also called training-serving skew) happens when the data or feature computation a model sees at inference time differs, even subtly, from what it saw during training. The model was never wrong about the training distribution — it's being asked questions from a different distribution without anyone noticing.

Common causes: a feature computed in a batch pipeline (e.g., Spark, using historical aggregates) is recomputed differently in a low-latency online service (e.g., a hand-written Java function using slightly different time windows); training data was collected before a schema change and silently has different units, defaults, or null-handling than production traffic; or training used labels that leak future information (e.g., a 'days since last purchase' feature computed with knowledge of a purchase that, at serving time, hasn't happened yet).

The insidious part is that train/serve skew usually doesn't crash anything. The model keeps making predictions — they're just quietly worse, and dashboards built on predicted-vs-actual accuracy may not catch it for days or weeks.

The feedback loop: predictions shape future data

Most production ML systems don't just consume data — they influence the data that will be collected next, because the model's own predictions change user or system behavior. A recommendation model that over-recommends a certain content category causes users to engage more with that category, which generates more training data confirming the model's own bias — a self-reinforcing loop with no external correction.

This is fundamentally different from a static supervised learning problem. A fraud model that flags transactions changes what gets manually reviewed, which changes what gets labeled as fraud, which changes what the next model version is trained to detect — potentially creating blind spots in exactly the areas the model is already weak. A credit model that denies a type of applicant removes those applicants from future repayment data entirely, making it impossible to ever learn whether the denial was actually justified.

Engineers need to design explicitly against this: holdout groups who don't receive the model's influence (to collect unbiased ground truth), randomized exploration (as in bandit algorithms) rather than pure exploitation, and periodic audits of whether the training distribution is drifting toward the model's own blind spots rather than reality.

Traditional software vs. ML systems

Design implications

These four properties — data dependency, non-determinism, train/serve skew, and feedback loops — are why ML system design treats testing, monitoring, and versioning differently than regular software. You version datasets and feature logic, not just code. You monitor prediction distributions and input data distributions, not just error rates and latency. And you design holdouts and randomization into the system itself, because without them you cannot even measure whether the feedback loop is corrupting your own training signal.

What's next

These fundamentals set up the full picture of how an ML system actually gets built and kept alive in production. The next lesson, The Machine Learning Workflow & Lifecycle, walks through that end-to-end loop — from problem framing through retraining — and explains why it's a loop rather than a line.