By the 5Tech engineering team · Published September 2024 · Reviewed for OT-safety and control-scope language. This is an educational engineering guide; the market and benchmark figures are cited or labelled illustrative, and the examples are explicitly typed (industry example / illustrative).
Most edge-AI projects that stall don't stall on the model. They stall on the enclosure that couldn't take the heat, the update that bricked a node in the field, or the false alarms that taught operators to ignore it. Edge AI earns its place when a specific problem carries a measurable constraint — latency, availability, bandwidth, or data governance — and when the full operational and safety lifecycle behind it can be engineered and maintained. Absent that, it is a science project with a support contract.
The shift is real — but it isn't cloud versus edge
For years the reflex was to stream everything to the cloud and analyze it there. On a modern line that reflex breaks down: the data pouring off vision systems, PLCs, and IIoT sensors has outgrown what a plant can economically backhaul, and a subset of decisions carry constraints a round trip can't meet.
Three things changed at once: sensor data volumes rose sharply, GPU- and NPU-class silicon arrived in fanless industrially-rated packages, and a class of use cases emerged that benefits from on-machine response. Together they move the question from whether to process at the edge to which workloads genuinely belong there.
The goal is never to move everything to the edge — only the decisions that carry a real constraint the cloud can't meet.
Think in two layers, not one
The most common architecture mistake is treating "edge AI" as one thing. It is two layers with very different rules, and conflating them is how safety and timing get quietly compromised.
Deterministic control layer
PLCs, RTOS, and safety controllers. Real-time, bounded jitter, fail-safe, and — where required — functionally certified. AI does not replace this.
Edge intelligence & management layer
AI inference, fleet management with K3s (a lightweight Kubernetes that runs on the containerd runtime), telemetry, and cloud sync. It inspects, advises, and optimizes — and sits above the control layer.
The distinction matters because the reflexes that work in IT — rolling container updates, orchestration, "move fast" — can disrupt the timing and availability OT depends on. NIST's operational-technology guidance is explicit that acceptable delay and jitter are application-specific, that real-time requirements must be defined, not assumed, and that safety comes before availability, integrity, and confidentiality (NIST SP 800-82r3). So a model earns trust as monitoring or decision support first, then perhaps supervisory input — and it participates in a control loop only after deterministic-timing and functional-safety validation, alongside (not instead of) validated control logic. Its output should pass a validation gate — confidence and range checks — before it can influence anything, and on low confidence the system should take no action or move to a known-safe state.
Reference architecture. Field sensors feed a deterministic control layer and an edge-intelligence layer; the AI's output passes a validation gate before it can advise control, and low confidence drives a safe state. The cloud sits across an OT/IT trust boundary and is never in the real-time path; field and edge keep operating during a WAN outage. (Illustrative reference architecture, not a specific installation.)
A diagram is not a design. A real one also fixes who owns the authoritative process data, how timestamps are synchronized, how sensor data is buffered during a failure, how models are approved and versioned, how rollback works while disconnected, and how redundancy and high availability are handled — each of which deserves its own review.
The intelligence layer, component by component
Within that upper layer, a production system is a stack, and each layer constrains the ones above it. The model is the last thing you choose, not the first.
- Data ingestion — fieldbus adapters, OPC UA, high-speed cameras. Your latency and bandwidth budgets start here.
- Edge compute & OS — a fanless x86 or ARM node with GPU or ASIC acceleration. The OS is a real decision: a general-purpose Linux host, a containerized service, and an RTOS have very different timing guarantees, so match it to whether timing behaviour actually matters for the workload.
- Inference runtime — ONNX Runtime, TensorRT, or OpenVINO, which decide what model formats and quantization you can use.
- Workload management — K3s (or another fleet-management platform), which uses containerd as its underlying container runtime — for rollouts and rollbacks of non-safety analytics, never on the real-time control path.
- Cloud bridge — an MQTT or AMQP gateway that carries aggregated insight and the retraining loop, without shipping raw data off-site.
On-machine inference in practice: a robotic arm carrying its own vision camera and edge accelerator inspects a body panel at line speed.
Measure the benefits — don't assume them
The benefits are real, but they are conditional, and the usual phrasing hides that. Where inference runs locally, response times can fall from the hundreds-of-milliseconds range typical of a cloud round trip to the low-milliseconds range — but only end to end, and only for a given model, resolution, hardware, and network. The number that matters is sensor-to-decision latency at the 95th percentile, including acquisition, preprocessing, inference, and post-processing — never model-inference time alone. That headroom supports low-latency supervisory decisions; it does not, on its own, make a function deterministic or safe enough for a control loop. The more dependable wins are operational: the line keeps running when the WAN link drops, and because only results are backhauled, bandwidth demand typically falls.
So treat any single figure as a measurement to reproduce, not a spec. Before anyone signs off, ask for a comparison like the one below — with the full test conditions (model, resolution, hardware, batch size, network) attached — on your data, your line. The numbers here are illustrative, not a customer result; the point is the shape of the evidence.
| Metric (end-to-end, stated conditions) | Cloud baseline | Edge — illustrative |
|---|
| Sensor-to-decision p95 latency | 240 ms | 18 ms |
| Raw data moved per day | 1.2 TB | 36 GB |
| Inspection accuracy (same test set) | 96.8% | 96.3% |
| Operation during WAN outage | Degraded | Continues locally |
Keeping data on-site genuinely reduces exposure and can simplify residency questions — but it does not, by itself, deliver compliance. An edge deployment still needs governance around access control, retention, audit logging, encryption, model provenance, the software supply chain, and remote-vendor access. Edge processing changes and often shrinks the compliance surface; it doesn't erase it.
What it costs — and when to say no
The costs that sink projects are operational, not computational. Hardware has to be qualified for temperature, vibration, and ingress. And models drift, so validation is a standing commitment — but the industrial version is staged, not the A/B testing of web apps: shadow mode first (the model runs but doesn't act), explicit acceptance criteria against a golden dataset, sign-off by a process engineer, change-control records with reproducible model artifacts, and automatic rollback triggers. Running divergent decision logic across machines without that discipline can quietly cost quality, uptime, or safety.
Security is the same story. Secure boot, hardware-backed identity, signed images, and network segmentation are necessary, not sufficient. A real deployment also has to answer certificate lifecycle and key rotation, least-privilege service identities, remote-vendor access, a software bill of materials and vulnerability management, recovery from a failed or interrupted update (including offline sites), audit logging and incident response, and the long tail of unsupported legacy equipment. Treat all of it as a dedicated OT security and change-control risk assessment — NIST is explicit that ordinary IT security measures can themselves disrupt OT timing or safety, not just protect it.
Because none of that is free, the decision should be a number, not a mood:
Annual value = reduced downtime + reduced scrap + labour saved + avoided cloud/network cost − ( hardware + integration + data labelling + model maintenance + security monitoring + validation + operator training )
Then price in the cost of being wrong — a false reject that halts a good line, a missed defect that ships. A technically excellent model can still have negative value if its false alarms interrupt production often enough that operators simply switch it off.
Fit the model to the floor
When AI is the right tool, a model that runs comfortably on a data-center GPU rarely fits an edge node unchanged. Quantization, pruning, and distillation — usually in combination — narrow the gap, though each trades some accuracy that has to be measured, not assumed; the result is then compiled for a specific runtime that maps it onto the target silicon.
Hardware and model are chosen together, not in sequence. GPUs bring flexibility and mature tooling; NPUs win on performance-per-watt for fixed models; FPGAs deliver deterministic latency at the cost of effort. The right answer falls out of the model, the latency budget, the power envelope, and the environment — the co-design at the heart of our machine-vision work.
A pattern worth copying: predictive maintenance
A common industry pattern (not a 5Tech customer result) is vibration-based condition monitoring: edge nodes on rotating equipment sample vibration and run a local model to flag early bearing wear as an advisory signal to maintenance, never a trip wired into the control loop. It is a good first use case because it is advisory, bounded, and measurable.
But a working system is far more than a model on a vibration stream. It depends on sensor type and mounting, anti-alias filtering, window length and overlap, handling of variable speed and load, and — hardest of all — honestly-labelled failure examples, which are usually scarce and imbalanced. Define the acceptance criteria up front (detection precision and recall, a false-alarm budget, minimum warning lead time), decide how it integrates with the maintenance system, and ask first whether a spectral-envelope or threshold detector already solves it more cheaply. For context on the size of the prize, McKinsey and Deloitte report meaningful downtime and uptime gains across programs — industry ranges to test against, not to assume.
What to remember
- Edge AI is a systems problem; the hard parts are enclosure, updates, drift, security, and false alarms.
- Split the architecture: validated deterministic control below, advisory AI above, with a validation gate between them.
- Measure sensor-to-decision latency end-to-end, with test conditions — never quote inference latency alone.
- Make it a business case, price in false-alarm cost, and run a dedicated OT security review.
- If a threshold, SPC chart, or classical vision solves it, use that instead.
Where to start
Skip the platform decision. Pick one line and one measurable problem — a defect that escapes, a failure that keeps surprising you — write down the acceptance criteria, the validation plan, and the fallback, and prove that an edge model moves that number end-to-end, on your data, in your environment. A working pilot on a single press will teach you more than any architecture diagram.
On the horizon, edge intelligence is integrating more tightly with real-time networks such as TSN and EtherCAT, and with standards work: the OPC Foundation is working to make its 400-plus existing Companion Specifications accessible to AI systems — through retrieval and MCP-style interfaces — rather than publishing a single "AI companion specification." Adoption will look very different in a greenfield plant than in one running decades-old equipment, and techniques like federated learning bring their own demands: non-identically-distributed plant data, model governance, the risk of poisoned or privacy-leaking updates, and the same shortage of labelled failures. They reduce the hard parts; they don't remove them. The teams that win won't have the boldest roadmap — they'll be the ones who shipped one honest, measured pilot and scaled what actually held up.
References & further reading
- NIST SP 800-82r3 — Guide to Operational Technology (OT) Security: application-specific timing, real-time requirements defined not assumed, safety-first priority order, and how IT measures can disrupt OT.
- McKinsey & Company — IoT value set to accelerate through 2030: factory settings as the largest IoT value pool; predictive-maintenance impact.
- Deloitte — Predictive maintenance and the smart factory: uptime, cost, and planning-time ranges.
- Grand View Research — Edge AI market report: market size and growth outlook.
- NVIDIA TensorRT, ONNX Runtime, Intel OpenVINO: quantization workflows, accuracy considerations, and hardware-specific optimization.
- OPC Foundation: Companion Specifications and the initiative to make them accessible to AI systems.
The diagram is an illustrative reference, not a specific installation.
Send 5Tech one workflow, inspection task, sensor problem, robotics challenge, or prototype idea. We will review it and suggest a practical next step. Start My Free First Phase — a free first review of one idea. If it is not practical, we will tell you.