BlogIoT and edge AI

Edge AI: running models on the device, next to the sensor

Edge AI means running a trained model on the device that collects the data, or on a gateway beside it, instead of calling a cloud service for every prediction. It is the right choice when the answer is needed in milliseconds, when the connection is unreliable or costly, or when the raw data should not leave the site. Most products end up with a split: small models on the device and heavier work in the cloud.

What edge AI is and what it is not

Edge AI is a decision about where inference happens. A model is trained somewhere with plenty of compute, usually the cloud, then shrunk and shipped to a device that sits next to the sensor: a camera on a production line, a vibration node bolted to a motor, a phone, a gateway in a plant room. The device runs the model on its own processor and acts on the result locally.

It is not a product you buy, and it does not make the cloud go away. The word covers everything from a coin-cell sensor with a few hundred kilobytes of memory to a GPU server in a factory cabinet, and the engineering at those two ends has little in common. The useful questions are narrower: which decisions must be made on the device, what hardware can make them, and how you keep hundreds of deployed models correct after they leave the building.

Why run a model on the device?

There are four reasons, and a project usually needs only one of them to justify the extra work.

Latency. A cloud round trip costs tens to hundreds of milliseconds on a good day and seconds on a bad one. A camera that rejects faulty bottles at ten per second has less than 100 ms to capture, decide and fire the reject gate. That decision cannot wait for a network.

Connectivity. Vehicles, farms, ships, mines and basements lose signal. If the product must keep working when the link drops, the model has to be on the device. Design for the offline case first and treat connectivity as a bonus.

Privacy. Video of a hospital ward or audio from a living room is easier to defend when it never leaves the device. Sending a count, an event or a blurred crop instead of the raw stream removes a whole class of data protection questions.

Bandwidth cost. A single compressed 1080p camera stream runs to a few megabits per second. A hundred of them on a cellular plan is a bill nobody approved. A model at the edge turns continuous video into a few kilobytes of events.

The case against is just as concrete. On-device models are smaller and less accurate than what a data centre can run, every hardware variant needs its own build and tests, and updating a model in the field is harder than redeploying a server. If your inputs are small, your users are online and a second of delay is acceptable, call an API and spend the effort elsewhere.

Which hardware fits which workload?

Hardware comes in tiers, and the tier decides what kind of model is possible. This is the table we use to size a first prototype.

Tier Typical hardware Memory and power Workloads that fit
Microcontroller Arm Cortex-M or ESP32 class, no operating system Kilobytes to a few megabytes, milliwatts Keyword spotting, vibration anomaly detection, simple motion or gesture classification
Microcontroller with a neural accelerator Cortex-M paired with an NPU such as Arm’s Ethos-U family A few megabytes, still battery-friendly Person or object detection at low resolution, audio event classification, small transformer models on the newest parts
Single-board computer with an accelerator Raspberry Pi 5 with an AI HAT+, or a similar Linux board with an NPU Gigabytes, a few watts One or two camera streams of detection or pose estimation, small local language models on boards built for them
Edge GPU module NVIDIA Jetson Orin and Thor families 4 GB to 128 GB, roughly 7 W to 130 W Multi-camera vision, vision-language models, robotics, speech pipelines
On-site server Industrial PC or rack server with one or more GPUs Limited by the cabinet’s power and cooling Dozens of streams, larger language models, aggregation across a site

Some reference points from vendor documentation. Google says the core of LiteRT for Microcontrollers fits in 16 KB on a Cortex-M3 and needs no operating system. Arm describes its Ethos-U85 accelerator as supporting transformer networks natively. Raspberry Pi’s AI HAT+ comes in 13 and 26 TOPS versions for vision, and the AI HAT+ 2 adds 8 GB of onboard memory for local language models. NVIDIA’s Jetson modules run from the Orin Nano at 7 W to 25 W up to the AGX Thor at 40 W to 130 W.

Two rules save time. First, prototype one tier above where you expect to land, prove the accuracy, then squeeze down. Going the other way round means weeks fighting memory limits before you know the idea works. Second, check the dull things early: operating temperature, how long the part will stay in production, and whether the accelerator supports every layer in your model. An unsupported operation falls back to the CPU without warning, and 30 frames per second becomes four.

How do you make a model small enough?

Three techniques do most of the work, usually in this order.

Quantisation stores weights and activations as 8-bit integers instead of 32-bit floating-point numbers. Google’s LiteRT documentation puts full integer quantisation at about four times smaller and three or more times faster, and it needs a few hundred representative inputs to calibrate the value ranges. Microcontrollers and most NPUs require it. Accuracy usually drops a little, and the drop is not spread evenly: rare classes and borderline cases suffer first. Check per-class results, not the average.

Pruning removes weights or whole channels that contribute little. It pays off only when the runtime or the chip can exploit the gaps. Removing entire channels gives a smaller dense model that any runtime handles. Scattering zeros through the weights often saves nothing in practice.

Distillation trains a small student model to copy the outputs of a large teacher. It is the most work and often the biggest gain, because the student learns from the teacher’s graded predictions and not only from the labels. It is also how most compact language models are made, which we cover in small language models on device.

Before any of these, choose an architecture designed for small devices. Compressing a large model by a factor of fifty is harder than starting from one that was meant to be small.

Measure on the target device after every step. Accuracy on a laptop and latency on a development board are both the wrong numbers.

The toolchains people actually use

Names change often in this area, so these were checked against vendor documentation at the time of writing.

  • LiteRT is Google’s runtime and the successor to TensorFlow Lite. It converts models from PyTorch, JAX and TensorFlow into the .tflite format and runs them with CPU, GPU and NPU acceleration. LiteRT for Microcontrollers is the bare-metal variant.
  • ExecuTorch is PyTorch’s on-device runtime, which its documentation positions for everything from high-end phones to constrained microcontrollers. If your team trains in PyTorch, it is the shortest path.
  • ONNX Runtime runs models exported to the ONNX format and hands the work to hardware through execution providers: TensorRT and CUDA for NVIDIA, OpenVINO for Intel, QNN for Qualcomm, CoreML for Apple. One model file can serve several targets.
  • Vendor compilers sit underneath all three. Each accelerator has its own, and you will use at least one. This layer is where most unexplained failures happen.

Our preference is to keep the training framework and the deployment runtime as close together as possible, and to run the conversion step in continuous integration. A model that no longer converts should fail the build, not the release.

Updating and monitoring a fleet

A model on a device is software in the field. It will need replacing because the world it was trained on changes: new product variants, a dirty lens, winter light.

Over-the-air model updates

Treat the model as a versioned artefact, separate from the firmware, so that it can change monthly while the firmware changes twice a year. Each release carries a manifest along these lines:

model: seal-inspector
version: 2.4.1
sha256: 9f2c41...e1
requires:
  firmware: ">=3.2.0"
  hardware: ["rev-c", "rev-d"]
  runtime: "litert-int8"
rollout:
  stages: [2, 20, 100]      # percent of devices
  rollback_if:
    crash_loop: true
    p95_latency_ms: ">80"

Sign every package and verify the signature on the device. Keep the previous model on disk and switch back automatically if health checks fail. Roll out to a small group before the rest. For the operating system itself, use a dual-partition scheme in which the bootloader returns to the old image if the new one never confirms itself, which is how tools such as Mender work. Where bandwidth allows, run the new model in shadow beside the old one for a few days before switching.

For connected products sold in the EU this is becoming a legal matter as well as good practice. The Cyber Resilience Act, whose main obligations apply from 11 December 2027, expects manufacturers to handle vulnerabilities throughout a product’s lifecycle.

Monitoring models you cannot see

You cannot look at the inputs of a thousand devices, so each one has to report on itself. Send the model version, inference latency, the distribution of predicted classes and confidence scores, cheap input statistics such as image brightness or audio noise floor, and device temperature and memory. Where privacy rules allow, upload a small sample of raw inputs weighted towards the low-confidence ones. A shift in the confidence histogram is the earliest sign of drift you will get, and those samples become the next training set once they are labelled.

Take a bottling line that inspects cap seals at ten bottles a second. A camera and an edge GPU module sit at the line. An 8-bit detection model returns a verdict well inside the 100 ms budget and drives the reject gate directly, so nothing depends on the network. Overnight the device uploads every rejected image plus a small random sample of accepted ones. In the cloud a larger model re-checks them and people label the disagreements. When a new cap colour arrives and confidence sags on one line, the dashboard shows it within a shift. A retrained model goes to two lines first, then to the rest.

The edge and cloud split

Pure edge and pure cloud are both rare. The common shape has three layers.

  • On the device: the time-critical decision, filtering, and anything that keeps raw data local.
  • On a gateway or site server: aggregation across devices, models that need more memory, buffering when the uplink is down.
  • In the cloud: training, the model registry, fleet management, long-term storage, dashboards, and large models for the cases the device was unsure about.

That last pattern is worth copying. A small model handles the clear cases on the device and passes the uncertain ones upward, so the expensive model sees only a fraction of the traffic. Our guide to predictive maintenance with IoT and AI uses the same layering. The cloud half is ordinary MLOps: registries, pipelines, evaluation and cost control.

Decide per decision, not per product. List each inference the product makes with its latency budget, data sensitivity and frequency, and the split usually falls out of the list.

Where to start

A first week that tells you something:

  1. Write down the latency budget, the worst-case connectivity, what data may leave the site and the monthly data allowance. If none of them forces the edge, stay in the cloud.
  2. Pick one model and one candidate device a tier above your target. Get the unoptimised model running on it end to end, from sensor to action.
  3. Quantise it, measure per-class accuracy and latency on the device, and decide whether a smaller tier is realistic.
  4. Sketch the update path and the telemetry before a second device exists. Retrofitting them to a deployed fleet is the expensive way to learn.

The first decision is not the chip. It is which decision has to be made locally and how wrong it is allowed to be. Choosing small and on-device models follows from that, and so does the hardware. This is the work our IoT and edge AI team does, from sensor choice to fleet updates.

Questions

Asked often.

What is edge AI in simple terms?

Edge AI means running a trained machine learning model on or near the device that collects the data, such as a camera, a sensor node or a gateway, instead of sending the data to a cloud server for every prediction. Training usually still happens in the cloud. Only inference, the act of making a prediction, moves to the device.

Can a microcontroller really run a neural network?

Yes, for small models. Keyword spotting, vibration anomaly detection and simple motion classification run on 32-bit microcontrollers with a few hundred kilobytes of memory, using runtimes such as LiteRT for Microcontrollers or ExecuTorch. Language models and multi-camera vision need a Linux-class board with a GPU or a neural accelerator, and a lot more power.

Does edge AI remove the need for the cloud?

Rarely. Most products end up with a split. The device makes the decisions that cannot wait or that must stay on site, and the cloud handles training, fleet management, long-term storage and the larger models used on the cases the device was unsure about. Plan both sides from the start, including what happens when the link is down.

How do you update a model on devices already in the field?

Treat the model as a versioned file that ships over the air with the same care as firmware: signed packages, a staged rollout to a small group of devices first, health checks after installation and an automatic return to the previous version if something goes wrong. Keep the model separate from the base firmware so it can change more often.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.