Model Serving & Inference Systems

Serving is a different engineering problem than training

Training optimizes for throughput over large, static datasets with no latency requirement — a training job that takes six hours is usually fine. Serving optimizes for responding to individual (or small batches of) requests fast enough that a user or downstream system isn't kept waiting, often with strict SLAs measured in milliseconds. The same model artifact can require completely different infrastructure depending on which of these two modes it's running in, which is why 'serving' is its own engineering discipline rather than an afterthought to training.

Batch vs. online inference

Batch inference runs predictions on a large set of inputs at once, on a schedule, with no individual request waiting on the result — e.g., scoring every customer for churn risk overnight, or generating recommendations for all users once a day. It optimizes purely for throughput: you can use large batch sizes, maximize GPU utilization, and tolerate latency of minutes or hours because nothing is waiting in real time.

Online inference (also called real-time or synchronous inference) responds to individual requests as they arrive — a fraud check during checkout, a recommendation shown on page load — and is bound by strict latency requirements, often under 100ms end to end. This forces different tradeoffs: smaller batch sizes (or batch size 1), the model kept warm in memory rather than loaded per request, and infrastructure that can autoscale with unpredictable traffic rather than a fixed, predictable nightly job.

A hybrid pattern, micro-batching, groups a handful of near-simultaneous online requests into a small batch inside the inference server to get some of batch inference's throughput benefit without violating latency SLAs — common in LLM serving, where requests arriving within a few milliseconds of each other can share a GPU forward pass.

Batch vs. online inference

The latency/throughput tradeoff

Latency and throughput pull against each other at almost every layer of an inference system. Larger batch sizes improve GPU utilization and raise total throughput, but each request in the batch has to wait for the whole batch to be assembled and processed, raising per-request latency. Model compression (see Model Compression) helps break this tradeoff rather than just shifting it — a smaller, quantized model lowers both latency and the compute cost per request simultaneously, which is why serving-constrained systems invest heavily in compression rather than just throwing more hardware at a large model.

The other major lever is hardware placement: keeping a model loaded on GPU memory avoids repeated load time but costs continuously, while loading on demand saves idle cost but adds cold-start latency — a tradeoff nearly identical to serverless cold starts in regular web infrastructure.

Model versioning and rollback

Every model artifact deployed to serving should be versioned and immutable — never overwrite a production model file in place. This is what makes rollback possible: if a new version regresses a key metric, serving traffic can be routed back to the previous version's artifact within seconds, exactly the way a bad code deploy gets rolled back, rather than requiring an emergency retrain.

A model registry (MLflow's registry, SageMaker Model Registry, or an internal equivalent) tracks which version is in which environment (staging, canary, production), what data and code produced it, and its evaluation metrics — giving the serving layer a single source of truth for 'what is actually running right now,' which matters enormously during an incident when every minute spent figuring out which model version is live is a minute not spent fixing the problem.

A/B testing and canary rollouts for models

Because offline evaluation metrics don't always predict real-world impact (see The Machine Learning Workflow & Lifecycle), new model versions are typically rolled out gradually rather than switched over entirely. A canary release sends a small percentage of live traffic (often 1-5%) to the new version while monitoring its metrics closely before increasing the share. Shadow deployment goes further by having the new model run on real traffic but discarding its output (or just logging it) without it ever affecting the user — letting you compare predictions against the live model with zero user-facing risk before any traffic is actually routed to it.

Formal A/B testing splits traffic between model versions and measures the actual downstream business metric (conversion, revenue, false-positive rate) rather than just the model's offline accuracy — because the only evaluation that fully accounts for the feedback loops discussed in ML System Design Fundamentals is one measured on live traffic.

What's next

Shipping a model safely is only half the job — it also needs to keep working correctly after launch, as the world it's serving keeps changing. The next lesson, MLOps: Monitoring & Maintaining Models in Production, covers data drift, concept drift, and how teams detect and respond to silent model decay.