BlogCloud and DevOps

Serverless pub/sub and queues: choosing a messaging service

Queues, pub/sub topics, event buses and streams are four different tools. A queue hands each job to one worker, a topic copies each message to every subscriber, an event bus routes by content, and a stream keeps an ordered log that many readers can replay. Choose by the guarantee you need (order, fan-out, replay), then make every consumer idempotent, because in practice all of them deliver at least once.

Four things that get called messaging

Managed messaging services let one piece of code hand work or news to another without both being awake at the same moment, and there are no brokers to patch. The confusion starts because four different designs share one vocabulary.

A queue holds jobs. Each message goes to one worker out of a pool, is hidden while that worker has it, and is deleted when the worker confirms. If the worker dies, the message reappears.

A pub/sub topic broadcasts. A publisher sends one message and every subscription receives its own copy. The publisher does not know who is listening.

An event bus is a topic with a router in front. Subscribers register filters on the content of events, such as orders above a certain value from one region, and the bus delivers the matches. Buses also carry events emitted by cloud services and SaaS products.

A stream is an append-only log divided into shards or partitions. Records stay for a retention period whether or not anyone has read them. Each consumer keeps its own position and can rewind.

Take an online shop. When an order is placed, the card must be charged, stock reserved, an email sent and the analytics store updated. The standard answer is one order.placed event published to a topic that feeds a separate queue for each consumer. The clickstream from the shop’s website belongs on a stream.

What does each one guarantee?

Queue Pub/sub topic Event bus Stream
Who receives a message One worker from a pool Every subscription Every subscriber whose filter matches Every consumer group, each at its own position
Removed when A worker acknowledges it Each subscription acknowledges its copy It is delivered, unless the bus retains events Retention expires
Order Best effort, or per group in FIFO variants Best effort, or per key where offered Usually none Strict within a shard or partition
Replay No Sometimes, by seeking a subscription back From an archive or built-in retention Yes
Scales by Adding workers Adding subscriptions Adding rules Adding shards or partitions

At-least-once is the default

Every service in this article delivers at least once in its standard mode. Standard SQS queues promise at-least-once delivery with best-effort ordering. Google Pub/Sub does the same by default, with no ordering guarantee. Cloudflare Queues say so too, and suggest a unique ID in each message for deduplication. Duplicates are rare in testing and routine in production, where they turn up during retries, deployments and network trouble.

Ordering costs throughput

Order is always scoped to a key: a message group in SQS FIFO, an ordering key in Pub/Sub, a session in Azure Service Bus, a partition key on a stream. Within the key, messages are handled one at a time, so a slow or failing message holds up everything behind it. Across keys there is no order at all.

It also has a price in speed. An SQS FIFO queue handles 300 API calls a second per action, or 3,000 messages a second with batching, unless you switch on its high-throughput mode. Ask for order only where the business needs it, and choose the narrowest key that satisfies it: per order before per customer, per customer before global.

Exactly-once, read carefully

The phrase appears on product pages and means less than it suggests. SQS FIFO queues discard repeated sends that carry the same deduplication ID. Exactly-once delivery in Google Pub/Sub stops redelivery once an acknowledgement has succeeded, but only on pull subscriptions, only within one region, and at the cost of noticeably higher latency.

None of these can protect you from your own worker crashing after it has charged the card and before it has acknowledged the message. The consumer has to be safe to run twice.

The managed services in each family

Family AWS Google Cloud Azure Cloudflare
Queue SQS, standard and FIFO Cloud Tasks; Pub/Sub pull subscriptions Storage queues; Service Bus queues Queues
Pub/sub topic SNS Pub/Sub Service Bus topics No separate service
Event bus EventBridge Eventarc Event Grid No direct equivalent
Stream Kinesis Data Streams; MSK for Kafka Pub/Sub with retention and seek Event Hubs K2, in public beta

Limits change, so treat the following as correct at the time of writing and check the documentation before you design around any of them.

  • SQS accepts messages up to 1 MiB and keeps them for 4 days by default and 14 at most. A received message stays hidden for a visibility timeout of 30 seconds by default, which can be raised to 12 hours.
  • EventBridge now has two kinds of custom bus. The newer Custom Event Bus retains every event for 1 to 365 days, lets a new subscriber start from a point in the past and offers ordered delivery within an event group. The older design with rules and targets is now labelled Classic and keeps nothing unless you add an archive.
  • Kinesis Data Streams keeps records for 24 hours by default and for up to 365 days.
  • Google Pub/Sub covers queue, topic and replay in one service. Cloud Tasks is the better fit when the sender needs to control the rate, schedule a delivery time or address one specific endpoint.
  • Azure draws the lines most clearly in its own comparison: Event Grid for discrete events, Event Hubs for streams, Service Bus for messages that need sessions, transactions, duplicate detection or dead-lettering. Storage queues are the basic option, with a 64 KB message limit.
  • Cloudflare Queues take messages up to 128 KB, keep them for up to 14 days and handle 5,000 messages a second per queue. K2, a durable log with retention of up to 30 days, is in public beta, so we would not yet build a system of record on it.

Dead-letter queues and idempotent consumers

Poison messages

A poison message is one that will never succeed: a malformed payload, a reference to a deleted record, a bug in the handler. Without a dead-letter queue (DLQ) it is retried until it expires, burning compute and burying real errors in the logs. In an ordered queue or a stream it also blocks everything behind it.

Configure a DLQ on every queue and subscription. In SQS a redrive policy sets maxReceiveCount, the number of times a message may be received before it is moved aside. Our defaults:

  1. Three to five attempts, with backoff, for errors that may pass.
  2. Straight to the DLQ, with no retries, for errors that never will, such as a payload that fails validation.
  3. A retention period on the DLQ longer than on the source queue. On standard SQS queues a message’s expiry runs from the moment it was first sent, so a short DLQ retention deletes the evidence early.
  4. An alarm on any message in the DLQ, routed to a person.
  5. A tested way to send messages back once the fault is fixed. SQS calls this redrive.

Making a consumer safe to run twice

Idempotent means a second run changes nothing. In order of preference:

  • Make the operation naturally idempotent. Set a status to paid and do not add one to a counter. Upsert by key and do not insert.
  • Record what you have processed. Write the message ID to a table with a conditional insert, in the same transaction as the work. If the insert fails, the work has been done already.
  • Pass an idempotency key downstream. Payment and email providers accept a key and return the original result on a repeat. Use the order ID, not a fresh random value for each attempt.

Check the visibility timeout as well. If a job takes 90 seconds and the timeout is 30, the queue hands the message to a second worker while the first is still busy. That is the commonest cause of duplicates we see, and slow jobs such as model calls make it more likely, which we cover in event-driven architecture for AI workloads.

How do the pricing models differ?

Three models are in use, and each rewards different habits.

Per request or operation. SQS bills each API call and counts every 64 KB of payload as a request. Cloudflare Queues bill each 64 KB written, read or deleted, so a message that succeeds first time costs three operations and every retry adds a read. Batching cuts the bill, since one SQS request can carry ten messages. Polling an empty queue costs money, so use long polling.

Per volume of data. Google Pub/Sub charges for the bytes published and delivered. Fan-out multiplies the charge, because every subscription is another delivery.

Per unit of capacity. Streams and premium tiers are sized in shards, throughput units or messaging units. You pay for them whether or not traffic arrives, unless the service has an on-demand mode, and longer retention costs extra.

Two habits help on all three. Keep payloads small by putting the document in object storage and its key in the message. And watch retries, which are billed like first attempts.

At low volume every option is cheap, and the engineering time spent choosing costs more than the service. Price starts to matter at hundreds of millions of messages a month, and by then you will have real traffic to measure.

Which one for which job?

You need to Use Why
Run a background job once per request: send an email, resize an image, call a model A queue One worker per job, with retries and a DLQ built in
Let several systems react to the same business event A topic that fans out to one queue per consumer Each consumer fails and retries on its own
Route events from many sources by their content, including cloud and SaaS events An event bus Routing is configuration, not code
Process updates to one entity strictly in order A FIFO queue or a stream, keyed by entity Order within the key, parallelism across keys
Ingest telemetry or clickstream at a high rate for several readers A stream Low cost per record at volume, and replayable
Reprocess history after fixing a bug A stream or a bus that retains events Only a log can replay
Call an HTTP endpoint at a controlled rate or at a later time A task queue with rate limits and scheduling The sender controls dispatch
Push updates to browsers or devices None of these Use WebSockets, server-sent events or MQTT

Device telemetry is a special case with its own protocol in front of the stream, which we describe in building an IoT data pipeline for AI.

The mistakes we see

Using a queue as a database. Teams peek into a queue to find the status of an order, or rely on long retention as the record of what happened. Queues cannot be queried and their messages expire. Put the state in a database table and let the queue carry only the ID of the job.

Ignoring poison messages. No DLQ, or a DLQ nobody looks at. The symptom is a steady background of errors that everyone has learned to scroll past.

Assuming order. A standard queue delivers mostly in order at low volume in development, and then stops doing so under load and retries. Either remove the need, by giving each update a version number and discarding anything older than what is stored, or pay for ordering with a FIFO queue or a keyed stream.

Watching depth instead of age. A queue with 10,000 messages that drains in a minute is healthy. A queue with 50 messages where the oldest has waited an hour is not. Alarm on the age of the oldest message.

Where to start

In the first week:

  1. Write down each message your system sends and label it: a job for one consumer, or a fact that several may care about.
  2. Start with one queue and one DLQ for the jobs, on the cloud you already use. Add a topic in front when a second consumer appears.
  3. Make the first consumer idempotent and prove it by delivering the same message twice in a test.
  4. Set the visibility timeout above your slowest job, and add alarms on message age and on the DLQ.
  5. Leave streams alone until you can name the replay or ordering requirement that needs one.

The first decision is whether each message is a command or an event. The service usually follows from that answer. Queues, buses and the workers behind them are part of the cloud and DevOps work we do, often alongside the data pipelines they feed.

Questions

Asked often.

What is the difference between a message queue and pub/sub?

A queue holds work for one logical consumer: each message is handed to one worker and deleted when that worker confirms it. Pub/sub delivers a copy of each message to every subscriber, so several independent systems can react to the same event. Most real designs use both, with a topic that fans out into one queue per consumer.

Is exactly-once delivery real?

Within limits. SQS FIFO queues drop repeated sends that carry the same deduplication ID, and Google Pub/Sub pull subscriptions can stop redelivery of a message once its acknowledgement has succeeded, inside one region. Nothing can stop your own code failing halfway through and running again. Write consumers so that processing the same message twice has no extra effect.

When should I use a stream such as Kinesis or Kafka instead of a queue?

Use a stream when the data is a continuous series that several consumers read at their own pace, when order within a key matters, or when you need to replay history through new code. If each message is a job to be done once and then forgotten, a queue is simpler, costs less at low volume and needs no decisions about shards or partitions.

Do I need a dead-letter queue?

Yes, on every queue and subscription that feeds code you wrote. Without one, a message that always fails is retried until it expires, which wastes compute and hides the fault. With one, the message moves aside after a set number of attempts, an alarm fires, and you can inspect it, fix the consumer and send it back.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.