ML Cost Optimization

The training-cost illusion

Training cost is the number that shows up in headlines — a large model's training run can cost anywhere from thousands to tens of millions of dollars in compute. It's also a one-time cost (or at most a periodic one, if you retrain regularly). Inference cost, by contrast, is incurred on every single prediction the model ever serves, for as long as it's in production. For any model serving meaningful traffic over months or years, cumulative inference cost overtakes training cost quickly, often within weeks — a model trained for $50,000 that serves 10 million predictions a day at even a fraction of a cent each will burn through that training cost many times over before the model is retired.

This asymmetry means cost optimization effort is almost always better spent on inference efficiency than on training efficiency, unless you're retraining extremely frequently or training unusually large models relative to your traffic volume.

Training cost vs. inference cost at scale

Spot instances: training's best friend, inference's risky bet

Spot (preemptible) instances offer compute at a steep discount — often 60-90% off on-demand pricing — in exchange for the provider being able to reclaim the instance with little notice. Training jobs are usually a great fit: checkpointing periodically means a preempted job loses at most a few minutes of progress and resumes on a new instance, and large distributed training runs can absorb the occasional node replacement without much disruption.

Inference serving is a much worse fit for spot instances in most cases, because a reclaimed instance mid-serving means dropped or delayed requests for real users right now, not a resumable background job. Spot instances for inference only make sense when the serving layer has enough redundancy and fast failover that losing any single instance is a non-event — which is achievable, but requires deliberate architecture rather than a drop-in substitution for on-demand capacity.

Right-sizing compute for inference

The most common and most fixable source of inference waste is simply running the wrong hardware for the job: a large, over-provisioned GPU instance serving a model that would run comfortably — and far more cheaply per request — on a smaller GPU, a CPU instance, or even a specialized inference accelerator. Right-sizing starts with profiling actual latency and throughput requirements against a range of instance types, not assuming the instance type used for training is the right one for serving.

Beyond instance selection, three techniques reliably cut inference cost: batching requests together to improve hardware utilization (especially effective for GPU inference, where per-request overhead dominates at low batch sizes), model compression (quantization, pruning, distillation) to reduce the compute and memory footprint per prediction, and caching predictions for repeated or near-duplicate inputs, which eliminates compute entirely for a meaningful fraction of traffic in many applications.

What's next

Cost efficiency often intersects with portability — being able to run a model on whatever hardware is cheapest at the moment, regardless of which framework trained it. Continue to ONNX & Model Interoperability to see how a common model format avoids locking a model (and its cost profile) to a single framework or deployment target.