Lesson 45 · Building systems

Message queues

A message queue sits between the part of a system that creates work and the part that does it. Follow a shop that puts email jobs on a queue and two workers that take them off: see how acknowledgements keep jobs safe, why a crash can deliver a message twice (so handlers must be idempotent), how failing messages are retried and then parked in a dead-letter queue, and how a topic gives every subscriber its own copy.

12 min10 steps5 questionsBeginner

Fig 45 · Interactive

Loading figure…

or ← →

Check yourself

5 quick questions

Pick an answer to see why. Your score saves with your progress.

01A shop puts an 'email the customer' message on a queue instead of sending the email itself. What does it gain?

The queue decouples the two: the producer only needs the queue, and consumers work through messages at their own pace.

02When does a broker such as RabbitMQ delete a message a worker has taken?

Until the ack arrives the broker keeps the message. If the worker's connection drops first, the message is delivered again.

03Why must a handler for an at-least-once queue be idempotent?

After a crash before the ack, the message comes back. Checking a shared list of finished ids (or using an idempotency key) stops a second email.

04A message has failed on every one of its allowed tries. Where does a well-configured queue put it?

A dead-letter queue holds messages that cannot be handled, so they stop blocking the others and are not lost.

05An order event must reach both the email service and the stock service. What fits?

With publish/subscribe every subscriber receives a copy. A work queue shares messages out, one worker per message.

Key terms

Glossary

Producer: The part of a system that creates work and puts messages on a queue.
Consumer (worker): A program that takes messages off a queue and does the job each one describes.
Message queue: A store that holds messages in order until a consumer takes them, so producers and consumers never call each other directly.
Acknowledgement (ack): The consumer's signal that it has finished a message; only then does the queue delete it.
At-least-once delivery: A promise that every message is delivered, possibly more than once (for example after a crash before the ack).
Idempotent: Safe to repeat: doing it twice has the same effect as doing it once.
Dead-letter queue: A side queue for messages that failed too many times or cannot be handled, kept for a person to inspect.
Publish/subscribe (topic): A pattern where every subscriber to a topic receives its own copy of each message.

Fine print

Notes and sources

The code card uses Python's standard queue.Queue, which works inside one program; there task_done() plays the part of the ack. With a broker over the network the calls differ (RabbitMQ's pika client calls ch.basic_ack, SQS calls DeleteMessage), but the loop has the same shape.
The list of finished ids must live in shared storage such as a database, not in one worker's memory. A crash between sending and noting can still send twice, so careful systems write the id in the same transaction as the work, or pass it to the email service as an idempotency key.
How the queue keeps its copy differs: RabbitMQ holds a delivered message as unacknowledged and requeues it if the consumer's connection closes; Amazon SQS hides it for a visibility timeout and shows it again if it is not deleted in time.
The retry limit (3 here) is a setting, such as SQS maxReceiveCount or a RabbitMQ quorum queue delivery-limit. Waiting a little longer before each retry (exponential backoff), with some randomness (jitter), avoids hammering a service that is struggling.
The drawing reuses one queue as the topic. In RabbitMQ a fanout exchange copies each message into a separate queue per subscriber; in Kafka each consumer group reads the whole topic at its own offset.
Oldest first is the usual case but not a promise: with several workers, retries and redeliveries, messages can finish out of order. Amazon SQS standard queues are best-effort ordering; FIFO queues and Kafka partitions keep order.

Quiz Unlocks after the last step.