By the 5Tech engineering team · Published January 2023 · Written for ML engineers and embedded-ML leads. This is an educational engineering guide; every compression figure is either cited to its primary source or explicitly labelled illustrative, and the worked numbers are typed as examples, not 5Tech customer results.
Compression is not a free lunch. Quantization, pruning, and distillation each buy you a smaller, faster model in exchange for accuracy you have to spend — and the exchange rate depends entirely on your model, your data, and your target silicon. The single most expensive mistake in edge ML is treating a technique's accuracy impact as "negligible" because a blog post said so, rather than measuring it on a representative test set of your own. Everything below is in service of that one rule.
The trade you are actually making
A model trained on a data-center GPU (32-bit floating point, or FP32) rarely drops onto a 2 W system-on-chip unchanged. Compression closes that gap along three axes at once — model size, inference latency, and energy — by spending a fourth: task accuracy. The techniques differ only in how they make the trade and how much control you have over it.
The reason "negligible" is a dangerous word is that aggregate accuracy hides where the loss lands. A quantized detector can hold its overall top-1 or mean-average-precision (mAP) to within a fraction of a point while quietly getting much worse on the rare class you actually deployed it to catch. Averages are reassuring; the failure modes live in the tails.
The only "negligible" accuracy loss is the one you measured on your own test set — per class, and at the edges.
Quantization: PTQ versus QAT
Quantization maps FP32 weights and activations to a lower-precision integer format — most commonly INT8. Because INT8 arithmetic is roughly a quarter of the memory and is natively accelerated on almost every edge NPU, DSP, and modern CPU, it is usually the first lever you reach for. There are two ways to get there, and the difference matters.
Post-training quantization (PTQ) takes an already-trained model and quantizes it, using a small calibration set to estimate activation ranges. It needs no labels and no retraining, so it is fast to try. On robust convolutional networks the loss is often small — TensorFlow reports MobileNet v1 losing about 1% top-1 when fully integer-quantized on ImageNet (with 100 calibration images) (TensorFlow Model Optimization).
Quantization-aware training (QAT) simulates the low-precision arithmetic during training so the network learns to compensate. It costs a training pipeline and labelled data, but it usually recovers most of what PTQ loses. NVIDIA's own example is blunt about why this is not optional: on EfficientNet-B0, PTQ collapsed top-1 accuracy from an FP32 baseline of 77.4% to 33.9%, while QAT brought it back to 76.8% — within roughly 0.6 points of the original (NVIDIA TensorRT). NVIDIA's guidance: "QAT almost always produces better accuracy, and sometimes this is the only acceptable method."
The practical rule is start with PTQ, validate, and reach for QAT only if the measured drop exceeds your budget. That ordering is exactly what the TensorFlow and ONNX Runtime quantization guides recommend. Two implementation details decide whether PTQ succeeds: the first and last layers are often precision-sensitive and are best kept in higher precision, and your calibration set must actually resemble field data — calibrate on the wrong distribution and the "post-training" step silently poisons the model.
Pruning: structured versus unstructured
Pruning removes weights the network can do without. The critical distinction is not how you choose them but what shape of hole they leave — because that determines whether any hardware can exploit it.
Unstructured pruning
Zeroes out individual weights anywhere in the tensor. It reaches high nominal sparsity with little accuracy loss after fine-tuning — but the result is a scattered sparse matrix that most edge runtimes execute at dense speed. The zeros save disk; they rarely save latency unless you have a genuinely sparse-aware kernel.
Structured pruning
Removes whole channels, filters, or heads, yielding a smaller dense model that every runtime accelerates directly. NVIDIA's hardware-friendly middle ground, 2:4 sparsity (two of every four weights zero), keeps ResNet-50 at 76.2% vs a 76.1% dense baseline after retraining on Ampere sparse Tensor Cores.
That 2:4 result — and BERT-Large holding 91.9% in both dense and sparse form — is measured, but so is its ceiling: NVIDIA reports the end-to-end gain as roughly a 30% performance-per-watt improvement, not the 2× the "50% of weights removed" headline might suggest (NVIDIA). The lesson generalizes: a sparsity number is a storage claim until you name the runtime and hardware that turn it into a latency claim. Unstructured pruning on a device with no sparse-execution path buys you a smaller file and the same speed.
Distillation: train a smaller model to imitate a bigger one
Knowledge distillation (KD) takes a different route: instead of shrinking the trained network, you train a compact student to reproduce the output distribution of a large teacher. The idea, introduced by Hinton, Vinyals & Dean in "Distilling the Knowledge in a Neural Network" (2015, building on Caruana's 2006 model-compression work), is that the teacher's soft probabilities carry richer signal than hard labels — the relative scores across classes encode how the teacher generalizes. The authors used it to compress an ensemble into a single acoustic model for a commercial speech system.
Distillation is powerful because model size becomes an independent design choice: you pick the student architecture, so you pick the latency and memory footprint outright. But it has a hard limit — a student with too little capacity cannot absorb the teacher's function no matter how good the teacher is. And it introduces a subtle validation trap: a student can match its teacher closely and still be wrong, if the teacher was wrong. Validate the student against ground truth on your task, never against agreement with the teacher.
Compression is the bridge from data-center-scale training hardware to a fanless edge node with a fixed power and memory budget — but each step across that bridge spends measurable accuracy.
The techniques side by side
These families are usually combined — quantize a pruned student — because they trade along different axes. The table summarizes what each buys and, more importantly, what you have to check before you trust it.
| Technique | Size effect | Latency / compute effect | Accuracy effect | What you must verify |
|---|
| PTQ (INT8) |
~4× smaller (cited) |
~2–4× on INT8 HW (cited) |
Often <1% (MobileNet v1 ~1%) — but can collapse (EfficientNet-B0 77.4→33.9) |
Per-layer sensitivity; keep first/last layers higher-precision; calibration set matches field data |
| QAT (INT8) |
~4× smaller |
Same as PTQ at runtime |
Usually within ~1% of FP32; recovers PTQ losses (EfficientNet-B0 →76.8) |
Training cost, labelled data, longer pipeline; still validate the final export |
| Unstructured pruning |
Smaller file; high nominal sparsity |
Little unless runtime/HW is sparse-aware |
Recoverable with fine-tuning |
Does your runtime actually accelerate scattered zeros? (often no) |
| Structured / 2:4 pruning |
Smaller dense model; ~50% weights (2:4) |
~30% perf/W on Ampere (cited); dense-runtime friendly |
~Unchanged after retrain (ResNet-50 76.1→76.2) |
Hardware support (Ampere-class sparse cores); retraining step |
| Distillation |
Whatever the student is (you choose) |
The student's — a design choice |
Depends on student capacity and teacher quality |
Student has enough capacity; validate vs ground truth, not teacher agreement |
Cited figures link to their primary sources and were measured on specific models and datasets. Treat any single number as a hypothesis to reproduce on your model and hardware, not a guarantee.
Validate like you mean it
Every technique above ends at the same gate: measured accuracy on a representative test set, compared against the FP32 baseline, before the model earns a place on a device. "Representative" is doing real work in that sentence — it means the same class balance, lighting, sensor, and edge cases you will meet in the field, not a convenient validation split. Report the delta per class and on worst-case slices, not just the aggregate, and set an explicit accuracy budget up front so "good enough" is a number, not a feeling.
The compression pipeline as it should run: nothing reaches a device until it clears a validation gate against the FP32 baseline, and the gate reopens on every retrain, calibration change, or observed drift. (Illustrative reference pipeline, not a specific deployment.)
This is also where the safety scope has to be honest. On constrained devices that feed control, medical, or safety-relevant decisions, a compressed model earns trust as a monitoring or advisory signal first — validated against the safety case and worst-case classes, with a deterministic fallback — before it is allowed anywhere near a closed loop. Speed and a good average score are not the same as safe, and an aggregate accuracy that "held" can still hide a regression on exactly the class that matters.
When not to compress the big model
Compression is a means, not a goal, and sometimes the cheapest path skips it entirely. Because size is a lever you control directly, the honest first question is whether you need the large backbone at all.
Worth compressing? value = ( latency + memory + energy + power-envelope headroom gained ) − ( accuracy lost × cost-per-error + engineering + validation + re-validation on every retrain/drift )
Compression has a lifecycle, not a finish line
A validated compressed model is a snapshot, not a permanent result. The field data drifts, so accuracy that passed the gate in July can quietly fall below budget by winter. Treat the artifact operationally: version the exact quantization recipe, calibration set, and pruning mask alongside the weights so a result is reproducible; keep the FP32 baseline available for rollback; and re-run the validation gate after any retrain, calibration-data change, or runtime upgrade — a new version of TensorRT, ONNX Runtime, or the vendor NPU compiler can change numerics enough to move accuracy. Monitor confidence distributions and input drift in production so you find the regression before your users do, and secure the update path so a model swap can't brick a fielded device.
What to remember
- Compression trades accuracy for size, latency, and energy — the exchange rate is model-, data-, and hardware-specific, never a constant.
- PTQ is the fast first try; move to QAT when the measured drop exceeds budget (EfficientNet-B0 is why: PTQ 33.9% vs QAT 76.8%).
- A sparsity number is a storage claim until you name the runtime and hardware that make it a latency claim — unstructured pruning rarely speeds up dense-only runtimes.
- Distillation lets you choose the student's footprint outright, but validate the student against ground truth, not against agreeing with its teacher.
- Nothing ships without clearing a validation gate on a representative set — per class, at the edges — and the gate reopens on every retrain and drift.
- If a smaller architecture or a classical method already meets the target, that beats compressing a big model.
Where to start
Pick one model and one honest test set that looks like your field data. Write down the accuracy budget and the latency, memory, and power ceilings of the target device. Run PTQ first, measure the per-class delta, and only escalate to QAT, pruning, or distillation if the number demands it — re-measuring at every step. A single technique validated end-to-end on your hardware will teach you more than any compression leaderboard. If you want a second set of eyes on that trade-off for a specific edge deployment, talk to the 5Tech engineering team about a model-and-hardware co-design review.
References & further reading
- Han, Mao & Dally — Deep Compression (ICLR 2016): pruning + trained quantization + Huffman coding; 35× (AlexNet) and 49× (VGG-16) with no accuracy loss after retraining; 3–4× layerwise speedup, 3–7× energy efficiency.
- Hinton, Vinyals & Dean — Distilling the Knowledge in a Neural Network (2015): soft-target distillation of a teacher (or ensemble) into a compact student.
- TensorFlow Model Optimization — post-training integer quantization and the quantization-aware training guide: ~4× smaller, 2–4× CPU speedup; MobileNet v1 ~1% top-1 drop; start with PTQ, use QAT for accuracy.
- NVIDIA — Achieving FP32 accuracy for INT8 inference with QAT and TensorRT: EfficientNet-B0 FP32 77.4% / PTQ 33.9% / QAT 76.8%; "QAT almost always produces better accuracy."
- NVIDIA — Accelerating inference with sparsity (Ampere + TensorRT): 2:4 structured sparsity; ResNet-50 76.1%→76.2%, BERT-Large 91.9% both; ~30% performance-per-watt end-to-end.
- PyTorch quantization documentation and ONNX Runtime quantization guide: INT8 ~4× size reduction, 2–3× speedup; dynamic vs static quantization; QDQ default and calibration-data handling.
The compression-pipeline diagram is an illustrative reference, not a specific deployment.
Send 5Tech one workflow, inspection task, sensor problem, robotics challenge, or prototype idea. We will review it and suggest a practical next step. Start My Free First Phase — a free first review of one idea. If it is not practical, we will tell you.