Results From an On-Device LLM Pilot for a Mobile Client

A mobile client wanted an on-device model for a feature that summarized user notes offline — no network round trip, no server bill per request, and a genuine privacy pitch they could put in their App Store listing. On paper it's the right call. In practice, we spent six weeks learning exactly how much of a 7B model's judgment you lose once it's quantized down to fit on a mid-range phone.

The latency number was the easy win

First response token on-device landed around 180ms on a reasonably modern Android device, versus 600-900ms round-tripping to a cloud endpoint depending on network conditions. For a feature meant to feel instant, that difference alone justified the project. Server costs dropped to basically zero for that feature too, since every inference happened on the user's own hardware. Both of those numbers showed up in the pilot deck and both were true.

On-device inference buys you latency and privacy. It charges you accuracy, and the invoice doesn't show up until you test the long tail.

Where it actually hurt

The quantized model, four bits down from the full-precision version we'd prototyped with, started mangling anything with structure — bullet points collapsed into run-on sentences, numbered lists lost their numbers, and a notable chunk of summaries dropped a detail that mattered to the user even though the overall gist survived. None of this showed up in our first round of testing because we tested on clean, well-formed notes. The actual messy inputs real users produce — half-finished thoughts, mixed languages, voice-to-text artifacts — is where the degradation showed up.

• Clean, short inputs: quantized model performance was nearly indistinguishable from full precision.
• Long or messy inputs: noticeable degradation in structure retention and detail accuracy.
• Low-end devices: thermal throttling during inference made latency inconsistent, not just slow.
• Multilingual inputs: accuracy dropped further than we expected, especially for anything the base model hadn't seen much of.

What we shipped instead

We ended up shipping a hybrid: on-device for short, clean inputs where the quantized model's accuracy was fine, with a fallback to a cloud call for longer or messier inputs where accuracy mattered more than latency. It's not the clean single-path architecture anyone wanted to present, but it matched the actual accuracy curve instead of pretending the curve was flat. The lesson that stuck with me is that on-device AI pitches sell the latency number and bury the accuracy cost in a footnote, and if you don't measure that cost yourself on your own messy data, you'll find out from your users instead.