BlogIoT and edge AI

From sensor to model: building an IoT data pipeline for AI

An IoT data pipeline for AI moves readings through six stages: devices publish over MQTT to a broker, a queue or log buffers the stream, a processor validates and enriches it, a time-series store and a raw archive keep it, labelled events turn it into training data, and models read it in the cloud or on the device. Most of the effort goes into identity, timestamps, schemas and duplicates, not into the model.

What the pipeline has to do

A model can only learn from what reaches it. Between a sensor and a model sit a radio link that drops, a clock that drifts, firmware in four versions and a database somebody chose in a hurry. The pipeline is everything that takes a reading off a device, gets it to storage intact, attaches the context needed to interpret it and hands it to a model.

The requirement is narrower than it sounds. Every reading should arrive at least once, belong provably to one device, carry a timestamp you can trust and a unit you know, and be available to replay. Miss one of these and the model learns the pipeline’s faults.

Keep one example in mind. A firm maintains 1,200 commercial refrigeration units across 300 supermarkets. Each controller reports compressor current, pressures, cabinet temperature and door state every ten seconds, roughly ten million messages a day. The firm wants an alert within a minute when a door is left open, and a model that flags a failing compressor days ahead.

MQTT: topics, QoS and retained messages

MQTT is the usual transport for device telemetry. A device opens one outbound TLS connection to a broker and publishes small messages to named topics, and any client subscribed to a matching topic receives them. The current version is MQTT 5.0, an OASIS standard since 2019, and 3.1.1 is still common in the field. We specify 5.0 for new firmware, because message expiry and reason codes save effort later.

Topics

A topic is a path with levels separated by slashes. Subscribers may use two wildcards: + matches one level and # matches everything below. Put the device identity in the path, so that permissions can be written per device:

v1/{site}/{device_id}/telemetry
v1/{site}/{device_id}/state
v1/{site}/{device_id}/event/door
v1/{site}/{device_id}/cmd/+

We insist on a version prefix, because the layout will change, and on separate branches for telemetry, state and commands, because they need different permissions. A device may publish only beneath its own ID and subscribe only to its own command branch.

QoS levels

QoS Name What happens Use it for
0 At most once Sent once, never acknowledged High-rate readings where a gap does no harm
1 At least once Resent until acknowledged, so duplicates can occur Nearly everything
2 Exactly once A four-packet exchange for each message Rarely worth the cost

QoS applies to one hop, between a client and the broker, not from end to end. Two of the large managed brokers do not offer level 2 at all: AWS IoT Core supports levels 0 and 1 only, and so does the MQTT broker in Azure Event Grid. Our rule is QoS 1 for telemetry and events, with a device ID and a sequence number in every message so that the consumer can discard duplicates. That gives the effect of exactly-once on any broker.

Retained messages, wills and sessions

A retained message is the last message published on a topic, kept by the broker and handed to each new subscriber at once. Retain state such as firmware version or setpoint, never telemetry: a dashboard that subscribes on Friday and is handed Tuesday’s temperature will show it as current.

A will is a message the broker publishes on a device’s behalf when its connection drops without a clean disconnect, which gives you presence detection. Sparkplug, the Eclipse specification for industrial MQTT, formalises this as birth and death certificates and adds a fixed topic namespace and payload format. In a plant full of PLCs, adopt it instead of inventing conventions.

Persistent sessions let a broker queue messages for an offline subscriber, but the allowances are small: Azure Event Grid ends a session whose queue passes 100 messages or 1 MB, and AWS IoT Core expires sessions after an hour by default. The broker is not your buffer. The device is.

Brokers, gateways and ingestion

Use a managed broker unless you have a reason not to. Running a clustered EMQX, HiveMQ or Mosquitto yourself is real on-call work, and makes sense mainly when data must stay on site.

Gateways and buffering

Sensors that speak Modbus, Bluetooth or a serial protocol reach the broker through a gateway that translates and publishes for them. The gateway, or the device itself, is where data waits when the network is down: a store-and-forward queue on flash, sized for the longest outage you intend to survive and replayed in order with the original timestamps. Each controller in our example keeps 48 hours.

Identity

Each device gets its own credential, normally an X.509 certificate whose private key never leaves a secure element, and the broker maps it to a device ID and a policy. Plan rotation and revocation before the first unit ships, because a fleet whose certificates all expire on one day is an outage with a date on it. And take the device ID from the authenticated connection, never from the payload.

From broker to stream

A broker routes messages and keeps almost no history, so a rule forwards every message into a durable queue or a log such as Kinesis, Kafka, Pub/Sub or Event Hubs. Ten million messages a day is about 120 a second, which a managed queue with one consumer carries easily. A log earns its place when a second and third consumer appear, or when you need to replay history through new code. We compare the options in choosing a messaging service.

Plan for the reconnect storm. When a regional network recovers, every device reconnects and flushes its buffer together. Add random jitter to reconnect delays in firmware and let the buffer, not the database, absorb the burst.

What happens to late and duplicate data?

Duplicates will occur, because of QoS 1 and replayed buffers. Data will arrive hours after it was measured, and out of order.

  • Idempotent writes. Key each reading on device ID and sequence number, and upsert. A duplicate then overwrites itself.
  • Two timestamps. Keep measured_at from the device and received_at from ingestion. The gap between them is a useful health metric.
  • Event time, stated explicitly. Stream processors such as Apache Flink separate event time, when something happened on the device, from processing time, and use watermarks to decide when a window is complete. The door alert fires on whatever has arrived within a minute. Training features are recomputed after 48 hours, once the slowest buffer has drained.

Do not start with a stream-processing framework. One consumer that validates, deduplicates and writes is enough for most fleets, and Flink can wait until you need windowed joins across devices.

Clocks

Devices set their clocks from NTP or the gateway, and a microcontroller without a battery-backed clock wakes up believing it is 1970. A device should not publish until its clock is set, or should flag its readings and include a monotonic counter so that the server can reconstruct the time. Ingestion quarantines timestamps in the future or implausibly far in the past. Everything is UTC.

Storage, schemas and units

Use two stores. The hot store is a time-series database for recent data, dashboards and online features: TimescaleDB, InfluxDB and ClickHouse are the ones we meet most. The cold store is every raw message exactly as received, in object storage as compressed Parquet files partitioned by date. For AI the archive matters more, because it lets you recompute features after a bug and retrain on history you did not know you would need.

At this volume a single database partitioned by time is enough.

Schemas and units

Every message carries a schema name and version, a device ID, a sequence number and the firmware version. Use a compact binary format such as Protocol Buffers on constrained links and JSON elsewhere. Validate at ingestion and send failures to a quarantine store instead of dropping them.

Put the unit in the field name, as in suction_pressure_kpa, never in somebody’s memory. If firmware 2.3 reports pressure in psi and 2.4 in kPa, the model sees a step change and learns the firmware version. A sensor fault is an explicit status, not a magic number like -999 that will later be averaged.

Readings also need context the device does not send: model, install date, site, sensor type and calibration. That lives in an asset registry joined to the telemetry. Keeping it accurate is ordinary data engineering, and it decides whether a model generalises across units.

From events to training data

A supervised model, the usual kind in predictive ML, needs labels: this compressor failed on this date for this reason. Labels come from work orders, technicians’ notes and acknowledged alarms, not from sensors. Store them with an asset ID, a start and end time, a type and a source, and start before there is a model, because labels cannot be reconstructed from memory.

  • Make each label an interval. A compressor that failed on 3 March was degrading from mid-February.
  • Record defrost cycles and maintenance visits as events. A defrost looks like a fault to a model that does not know it is happening.
  • Split training and test data by device and by time, never by random row. Readings ten seconds apart are nearly identical, and a random split leaks the answer.

Model choices are covered in our guide to predictive maintenance.

Feeding models in the cloud and on the device

In the cloud, training reads the archive and online scoring reads features computed from the stream, such as rolling means and rates of change. Both must run the same feature code, or the model meets one distribution in training and another in production.

On the device, a small model runs on the controller or gateway, which is edge AI in its plainest form. It arrives as a signed artefact through the firmware update channel and reports its inputs and outputs back, so that it can be monitored and retrained. The door alert is a rule at the edge. The compressor model can run hourly in the cloud, because a warning measured in days does not need millisecond latency.

A reference architecture

Follow one reading. A device timestamps and buffers it, then publishes at QoS 1 to a broker that authenticates the device by certificate. A rule copies the message to a durable buffer. A consumer validates, deduplicates and enriches it, then writes to the hot store and the raw archive. Models train on the archive and score from the stream or on the device.

Stage Job Typical choices What breaks
Device or gateway Sample, timestamp, buffer, publish MQTT 5 client, flash queue, NTP Clock unset after reboot; buffer too small
Broker Authenticate, authorise, route Managed MQTT broker, EMQX, HiveMQ, Mosquitto Wildcard permissions
Ingestion buffer Absorb bursts, allow replay Kinesis, Kafka, Pub/Sub, Event Hubs, a queue Reconnect storms; no dead-letter path
Processor Validate, deduplicate, enrich A consumer function first, Flink later Processing time used for event time
Storage Hot store and raw archive A time-series database; object storage with Parquet Only aggregates kept; no retention policy
Registry and labels Asset context, ground truth Tables fed by the maintenance system Free-text labels with no asset ID
Model serving Score in cloud or on device Batch job, stream consumer, on-device runtime Feature code differs from training

Monitoring data quality

A model fails quietly when its data degrades, so monitor the data and not only the servers.

  1. Freshness and completeness: time since each device’s last message, and gaps in sequence numbers.
  2. Lateness: received_at minus measured_at.
  3. Duplicate rate and schema failures, by firmware version.
  4. Range and flatline checks: values outside physical limits, or a sensor that reports exactly 2.6 °C for six hours.
  5. Drift: this week’s distribution of each model input against the training set.

Alert at fleet level, because 1,200 separate alarms are noise. The flatline is the nastiest case: every dashboard stays green while the model is fed fiction.

Where to start

In the first week:

  1. Write the message contract on one page: topic layout, versioned schema, units, both timestamps and the sequence number.
  2. Decide how devices get their identity and who issues the certificates.
  3. Stand up a managed broker and one consumer that writes to the raw archive and a time-series table.
  4. Connect five real devices, not a simulator, and unplug the network from one overnight. In the morning check what arrived, in what order and with which timestamps.
  5. Create the label table and connect it to wherever maintenance work is recorded.

The first decision is the contract, because firmware in the field is the hardest part to change later. The model can wait a month. The archive cannot. Device firmware, gateways and the platform behind them are what our IoT and edge AI team builds.

Questions

Asked often.

Which protocol should IoT devices use to send data to the cloud?

MQTT is the default for telemetry from devices that have an IP connection. It keeps one outbound TLS connection open, adds little overhead to each message and provides topics, delivery levels and a will message for spotting dropped devices. Use HTTPS for occasional uploads of large files, and OPC UA or Modbus at the gateway to read existing industrial equipment.

Should I use MQTT QoS 2 to get exactly-once delivery?

Usually not. QoS 2 covers only the hop between one client and the broker, costs a four-packet exchange per message, and is not offered by AWS IoT Core or the MQTT broker in Azure Event Grid, which accept levels 0 and 1. Publish at QoS 1, put a device ID and sequence number in every message, and remove duplicates when you store the data.

Do I need Kafka for an IoT data pipeline?

Not at the start. A managed MQTT broker with a rule that forwards messages to a queue and one consumer will carry a fleet of a few thousand devices. Add a log such as Kafka or Kinesis when several independent consumers need the same data, when you want to replay history through new code, or when reconnect bursts would overwhelm the database.

What data should I keep for training models later?

Keep every raw message exactly as it arrived, in cheap object storage, with the device's timestamp, the time of receipt, the schema version and the firmware version. Keep maintenance records and other labels with asset IDs and time intervals. Aggregates alone are not enough, because you cannot recompute features or correct a unit error from an hourly average.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.