ML Deployment Paradigms: Cloud, Edge & Mobile

Where should inference actually run?

Once a model is trained, a question that's easy to defer and expensive to get wrong is: where does it run when a real request comes in? The default answer for most teams is 'a server in the cloud,' because that's where training happened and where the team already has infrastructure. But cloud inference is one point on a spectrum, not the obvious endpoint. The same model can be served from a data center, from a cell-tower edge node, or directly on the user's phone, and each choice trades off latency, cost, privacy, and reliability differently enough that it should be a deliberate architectural decision, not an inherited default.

Cloud inference: maximum capability, minimum control over latency

Running inference in a centralized cloud environment gives you the biggest models, the easiest updates (ship a new model version to one place), and the simplest monitoring (all traffic flows through infrastructure you control). The cost is a round trip: every prediction requires a network hop to a data center and back, which adds 50-300ms depending on geography even before the model runs. For a recommendation feed or a batch analytics job, that's irrelevant. For an AR filter that needs to track a face at 30 frames per second, it's disqualifying.

Edge inference: splitting the difference geographically

Edge deployment pushes inference closer to the user — a regional point-of-presence, a retail store's local server, a factory floor gateway — without putting the model on the end-user device itself. This cuts network latency substantially versus a centralized cloud region while still giving you more compute than a phone or sensor has, and it keeps model updates relatively centralized (you're updating dozens or hundreds of edge nodes, not millions of devices). Edge is the common choice for things like in-store computer vision, industrial sensor analytics, and CDN-adjacent content moderation, where the latency budget is tight but the device itself is too constrained or too numerous to manage individually.

On-device / mobile inference: zero network dependency

On-device inference runs the model directly on the user's phone, laptop, or embedded hardware, with no network call at all. This is the only option that works fully offline, gives the lowest possible latency (no round trip, period), and keeps raw user data from ever leaving the device — a meaningful privacy and compliance advantage for anything touching health, biometric, or financial data. The cost is that the model must fit the device's memory and compute budget, updates require shipping a new app build or a model download, and you lose easy visibility into exactly what's running where, since you can't instrument a user's phone the way you instrument your own servers.

Cost is not just compute cost

Teams often compare paradigms purely on compute cost per inference, but the fuller cost picture includes bandwidth (streaming video to the cloud for inference is expensive at scale, in a way running a compressed model on-device isn't), update distribution (a cloud model update is one deploy; an on-device update is an app store release cycle or a background model download that not every user will apply promptly), and failure mode cost (a cloud outage takes down every user simultaneously; a flaky on-device model only affects the users who have that broken binary, and a bad edge node only affects its regional footprint).

Cloud vs. edge vs. on-device inference

What's next

Choosing on-device or edge inference raises an immediate follow-up question: how do you fit a model into a few megabytes of RAM and a fraction of a watt of power? The next lesson, Edge AI & TinyML, covers the engineering discipline built specifically around that constraint.