Edge AI & TinyML

ML on a chip that costs less than a coffee

TinyML refers to running machine learning inference on microcontrollers — the class of chip found in a smart thermostat, a wearable fitness tracker, or an industrial vibration sensor. These chips typically have kilobytes, not gigabytes, of RAM, run on a coin-cell battery for months or years, and often have no operating system at all, just firmware. A server GPU might have 80GB of memory and draw 700 watts; a typical TinyML target has 256KB of RAM and draws milliwatts. That's not a smaller version of the same engineering problem — it's a different problem with a different toolchain, different model formats, and different failure modes.

Why this needs its own discipline

A cloud or mobile ML engineer who's never worked on microcontrollers tends to underestimate how binding the constraints are. It's not 'make the model a bit smaller' — it's routinely a 1000x reduction in available memory compared to a laptop, with no virtual memory to fall back on and no ability to simply allocate more RAM if the model doesn't fit. Every byte of the model's weights, every intermediate activation buffer, and the inference runtime itself all have to coexist in that same tiny memory footprint, often while leaving headroom for the sensor drivers and communication stack the device also needs to run.

The core techniques: quantization, pruning, and tiny architectures

Three techniques dominate TinyML model preparation. Quantization converts model weights and activations from 32-bit floating point to 8-bit (or even 4-bit or binary) integers, cutting memory footprint by 4x or more with usually modest accuracy loss. Pruning removes weights or entire neurons that contribute little to the model's output, shrinking the model directly. Architecture selection matters enormously here — rather than shrinking a large CNN, TinyML practitioners often start from architectures designed from scratch for extreme efficiency (depthwise separable convolutions, very shallow networks), because a naively shrunk large model usually loses more accuracy per saved byte than a model designed small from the start.

Power is the other hard constraint

Memory gets the most attention, but for battery-powered edge devices, energy per inference is often the binding constraint. A device sampling a sensor and running inference continuously will drain a coin-cell battery in days if inference is expensive, versus years if it's cheap enough to run only in short bursts. This pushes TinyML designs toward techniques like keeping the device mostly asleep and running a very cheap always-on model to decide when to wake a more expensive model — a two-stage 'cascade' that's common in wake-word detection (a tiny model listens for 'maybe someone said the wake word,' and only then does a larger model confirm it).

Mobile sits between TinyML and the cloud

Phones occupy a middle ground: far more capable than a microcontroller (gigabytes of RAM, dedicated neural processing hardware in many recent chips) but still constrained compared to a server, and still battery-limited. Mobile ML frameworks (Core ML, TensorFlow Lite, ONNX Runtime Mobile) apply the same quantization and pruning ideas as TinyML but with more headroom, and can lean on dedicated hardware accelerators that microcontrollers simply don't have. The engineering judgment call — how aggressively to compress, whether to use hardware acceleration, how to fall back gracefully on older devices — still has to be made explicitly per target device class.

Microcontroller TinyML vs. mobile edge ML

What's next

Getting a model small enough to fit a chip is only half the battle — the other half is making sure the hardware you're targeting, whatever its class, is actually well matched to the computation the model demands. The next lesson, System Balance: Matching ML Workloads to Hardware, looks at why raw processing power alone doesn't guarantee a fast system.