By the 5Tech engineering team · Published April 2021 · Written for embedded and firmware engineers. Every benchmark and market figure below is either cited to a primary source or explicitly labelled illustrative; the worked numbers are typed as illustrative and are not a 5Tech customer result.
On a microcontroller, the model is the easy part. The hard part is fitting it into tens of kilobytes of RAM with no operating system and no heap, holding a power budget measured in milliwatts, doing the arithmetic in fixed point, and still being able to update the thing safely five years after it shipped. TinyML — machine learning on microcontroller-class hardware — earns its place only when that entire budget is engineered together. And a surprising amount of the time, the honest answer is that a threshold or a filter beats the neural network.
Who this is for, and the one claim
This is for the firmware engineer and hardware lead who has been handed a trained model and a Cortex-M part and asked to "just run it on the device." Acronyms first, because they matter here: an MCU is a microcontroller unit; RAM here means on-chip SRAM measured in kilobytes; flash is the on-chip program store; INT8 and FP32 are 8-bit integer and 32-bit floating-point number formats; DSP is digital signal processing.
The single claim of this article: the accuracy of the model is rarely what decides whether TinyML ships. The device budget and the field lifecycle decide. Memory, power, deterministic timing, and the ability to patch a fleet you can no longer physically reach will make or break the project long before the last percent of accuracy does.
The question on an MCU is not "can the model be accurate enough?" It is "does it fit — in RAM, in milliwatts, and in the firmware release you still have to support years from now?"
The constraints, honestly
A model that runs comfortably on a workstation meets a different physics on a microcontroller. Four constraints dominate, and none of them is about accuracy.
- KB-scale memory. A typical Cortex-M4-class part has tens to a few hundred KB of SRAM and one to two MB of flash. The model's weights live in flash; its intermediate activations need a contiguous block of RAM — the tensor arena — that competes directly with your stack, heap, and DMA buffers. On these runtimes that arena is a fixed, statically sized region, because there is no dynamic allocation to fall back on (Google AI Edge). If it doesn't fit at build time, it doesn't fit.
- No OS, no heap. Many targets run bare-metal or a small RTOS. There is no virtual memory, no swap, and often no dynamic allocator you would trust in a ten-year product. Inference has to be a bounded, statically-provisioned task that shares the core with everything else the firmware must do.
- Fixed-point math. Plenty of MCUs have no floating-point unit, or only a single-precision one. That is why INT8 quantization — mapping FP32 weights and activations to 8-bit integers — is not an optimization on these parts, it is the entry ticket. It is also what the specialized kernels and accelerators are built to run.
- Milliwatt power budgets. An always-on model on a coin cell or an energy-harvested node has an energy budget per inference, not just a latency budget. This is measurable and worth measuring: MLPerf Tiny includes an energy track alongside latency and accuracy precisely because, at this scale, joules per inference is a first-class metric (MLCommons).
Two things share the core — keep them separate
On a shipped device the inference code is not the only thing running, and it is usually not the most important thing. Sampling deadlines, communications, a watchdog, and any safety or interlock logic all have to keep meeting their timing whether or not the model is confident, converged, or even present.
Deterministic firmware
Sensor sampling, control/interlock logic, comms, watchdog. Real-time deadlines that must be met every cycle. The model does not get to starve these — inference is scheduled around them, not the reverse.
TinyML inference
A bounded, statically-sized task that reads features, runs a quantized model, and emits an advisory result behind a confidence and range gate. On low confidence it does nothing, or falls back to a known-safe default.
This matters most when the device touches anything with a safety or regulatory dimension — a medical wearable, a smart meter, a machine guard. Speed and accuracy alone do not make an ML output safe to act on. Treat the model as monitoring or decision support that flags to a host or a human; let it become supervisory or closed-loop only after deterministic-timing and validation work that is out of scope for a model conversion. When in doubt, the safe state wins over the smart one.
Illustrative on-device pipeline. Everything lives in statically-sized memory with no heap; classical DSP does the feature extraction; the quantized model's output only becomes an action after a confidence-and-range gate, and low confidence falls back to a safe default. Model updates arrive as signed images in A/B flash slots so a bad update can roll back on a device that is offline. (Illustrative reference pipeline, not a specific product.)
The runtimes and silicon you'll actually pick from
You do not write these kernels yourself, and you should not. Verify names against the primary sources, because the field renames things: what many still call TensorFlow Lite for Microcontrollers (TFLM) is now LiteRT for Microcontrollers under Google's LiteRT umbrella.
- LiteRT for Microcontrollers — the interpreter most MCU projects start on. It is C++17, runs without an OS or dynamic allocation, and its core fits in about 16 KB on a Cortex-M3; total footprint grows with the operators your model needs (Google AI Edge).
- CMSIS-NN — Arm's optimized neural-network kernels for Cortex-M. In the original Arm benchmark on a CIFAR-10 CNN, CMSIS-NN kernels delivered about a 4.6× improvement in throughput and 4.9× in energy efficiency versus a baseline implementation — a specific model on a specific core, not a universal multiplier (Lai, Suda & Chandra, Arm, 2018).
- Arm Ethos-U microNPU — a neural processing unit (NPU) that offloads the heavy operators from the CPU. Arm states the Ethos-U55 is configurable from 32 to 256 multiply-accumulate (MAC) units, delivering 64–512 GOP/s at 1 GHz, and that paired with a Cortex-M55 it can offer up to a 480× uplift in ML performance over existing Cortex-M systems — a vendor headline for a best-case pairing, so treat it as a ceiling to verify, not a guarantee (Arm).
- Edge Impulse EON Compiler — instead of a generic interpreter, it compiles the model to C++ source. Edge Impulse reports this runs networks in 25–55% less RAM and up to 35% less flash than the interpreter path at the same accuracy (Edge Impulse).
- MLPerf Tiny (MLCommons) — not a runtime but the yardstick. It is the first industry-standard benchmark for ultra-low-power ML, scoring four reference tasks — keyword spotting, visual wake words, image classification, and anomaly detection — on accuracy, latency, and energy (MLCommons). When you need real, comparable numbers for a given part, this is where to get them rather than a slide.
The trade-off you can't cheat: memory vs accuracy vs latency
Getting a model onto the part is a chain of compressions — quantization first, then pruning, sometimes distillation — and each one trades away something you have to measure, not assume. Post-training quantization (PTQ) to INT8 is the big lever: it roughly quarters model size versus FP32 and lets the fixed-point kernels and NPUs do their work. Structured pruning removes whole channels to shrink further and cut latency. An NPU offload can collapse latency and energy without changing the model at all.
The table below shows the shape of that trade for a keyword-spotting-style network on a Cortex-M4-class part. Read the ordering, not the digits: the specific figures are illustrative, not a measured result. For real, comparable latency and energy on a named device, the credible source is a MLPerf Tiny submission for that model and board (MLCommons).
| Technique (illustrative; ~25k-param DS-CNN, M4-class, single inference) | Flash (weights) | Peak RAM (arena) | Latency | Accuracy vs FP32 |
|---|
| FP32 baseline (reference, off-device) | ≈100 KB | highest | reference | reference |
| INT8 post-training quantization | ≈25 KB | lower | faster | small drop — measure |
| INT8 + structured channel pruning | ≈15 KB | lower still | faster still | further drop — measure |
| INT8 + NPU offload (Ethos-U class) | ≈15 KB | similar | much lower | ≈ unchanged vs INT8 |
Two honest caveats. First, the accuracy loss from quantization and pruning is real and model-specific; validate every step against a representative dataset and a held-out set that reflects field conditions, not lab ones. Second, "latency" on a device means end-to-end — sensor read, feature extraction, inference, and the decision — reported at a high percentile, not the inference kernel in isolation.
When a neural net is the wrong answer
The most valuable thing a TinyML review can produce is sometimes the decision not to use TinyML.
Make it a number: value versus cost
Because none of this is free, the go/no-go should be arithmetic, not enthusiasm. On a shipped device the costs are dominated by things that have nothing to do with the model's cleverness:
Lifetime value = avoided connectivity/cloud cost + new or premium function + battery life gained + bandwidth saved − ( larger MCU/NPU BOM + fixed-point engineering + data collection & labelling + validation + OTA infrastructure + field maintenance + security )
Then price the cost of being wrong in a place you can't easily reach. A false positive that trips a good machine, or a false negative on a smart meter that misses a fault, carries a field cost — a truck roll, a warranty claim, a recall — that dwarfs the compute. A technically strong model can still be net-negative if its error rate generates callouts, or if a marginally larger part blows the bill of materials on a device you ship by the million. The unit economics of a mass-produced device are unforgiving in a way a cloud service never is.
TinyML lives here: a sensor and a microcontroller with kilobytes of RAM and a milliwatt power budget, running quantized inference with no operating system underneath it.
The part everyone underestimates: lifecycle in the field
A microcontroller product can live in the field for a decade, often with no reliable connection and no one to press a button. Deployment is the start of the work, not the end.
- Updates and rollback. Ship models — and firmware — over the air (OTA) into A/B (dual-bank) flash slots so a failed or interrupted update rolls back to the last good image without bricking a device you can't touch. On sub-megabyte parts, consider model-only or delta updates to fit the pipe. The update path is a safety feature, not a convenience.
- Provenance and security. A model on a device is firmware, and therefore attack surface. Secure boot with a hardware root of trust, a signed model whose hash is verified before inference, key rotation, a software bill of materials (SBOM), and vulnerability handling are baseline. None of it is optional on a connected product with a ten-year life.
- Versioning and reproducibility. Every deployed model needs a version, a signed artifact you can rebuild bit-for-bit, and a record of the data and quantization settings behind it. When a field failure arrives, "which model is on that unit, and how do we reproduce it?" must have an instant answer.
- Drift — with realistic telemetry. Sensors age, mounting shifts, environments change, and accuracy decays. On a constrained device you rarely get rich telemetry back, so plan for lightweight drift signals (confidence distribution, simple population-stability checks) and a defined trigger to capture data and refresh the model — not a fantasy of streaming everything home.
- End of life. Decide up front when a device stops getting model and security updates, and how it behaves after that — ideally by degrading to the deterministic firmware path it never depended on the model to begin with.
A prediction, scoped
Over roughly the next few years, and assuming current silicon trends hold, sub-milliwatt always-on inference should become ordinary on mainstream MCUs as microNPUs and better fixed-point support spread down the price curve — ABI Research's multi-billion-unit 2030 forecast points the same way (ABI Research). Benchmarks are maturing alongside the silicon: MLPerf Tiny added a streaming (wake-word style) benchmark in its v1.3 release, which better reflects always-on workloads. What will not change is the discipline: the winners will be the teams that sized the budget honestly, validated every compression step, and could still safely update a fleet years after shipping.
What to remember
- On an MCU the model is the easy part; memory, milliwatts, fixed-point, and the field lifecycle decide whether it ships.
- Everything is statically sized — no heap. The tensor arena competes with your stack and DMA buffers, and INT8 quantization is the entry ticket, not an optimization.
- Keep deterministic firmware separate from advisory inference; gate the model on confidence and range, and fall back to a safe default.
- Quote latency and energy from a MLPerf Tiny submission for the real device; treat any table (including ours) as illustrative until measured.
- If a threshold, FFT, or SPC rule meets the spec, use it — it is cheaper, deterministic, and far easier to certify than a neural net.
- Deployment is the start: OTA with A/B rollback, signed models, versioning, drift signals, and an end-of-life plan.
Where to start
Pick one function and one number. Write down the memory ceiling (arena plus stack plus buffers), the energy-per-inference budget, and the end-to-end latency target — before you touch the model. Then prove, on the actual part, that a classical detector can't already meet the spec; only if it can't does the neural network earn the RAM. Prototype the update-and-rollback path early, because it is harder than the inference and it is the thing that will still matter in year five. If you want a second pair of eyes on whether a problem needs a neural network at all — or on the memory, power, and update budget behind one — that is the kind of pilot our embedded team runs; start the conversation with 5Tech.
References & further reading
- Google AI Edge — LiteRT for Microcontrollers overview: ~16 KB core runtime on Cortex-M3; no OS, no standard C/C++ library, no dynamic allocation; supported platforms.
- Lai, Suda & Chandra (Arm), CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs: 4.6× throughput and 4.9× energy improvement on a CIFAR-10 CNN versus baseline.
- Arm — Ethos-U55 microNPU: configurable 32–256 MACs, 64–512 GOP/s at 1 GHz, and the up-to-480× uplift claim when paired with Cortex-M55.
- MLCommons — MLPerf Tiny inference benchmark: four reference tasks; accuracy, latency, and energy metrics; sub-100 KB reference models.
- MLCommons — MLPerf Tiny v1.3 streaming benchmark (2025): a wake-word-style streaming task for always-on workloads.
- Edge Impulse — EON Compiler: 25–55% less RAM and up to 35% less flash than the interpreter path at equal accuracy.
- ABI Research — TinyML device shipments: forecast growth from ~15 million (2020) to 2.5 billion (2030).
The on-device pipeline diagram is an illustrative reference, not a specific product.
Send 5Tech one workflow, inspection task, sensor problem, robotics challenge, or prototype idea. We will review it and suggest a practical next step. Start My Free First Phase — a free first review of one idea. If it is not practical, we will tell you.