Queues let you decouple producers from consumers, absorb spikes, and do slow work off the request path. Knowing when to add one — and SQS vs Kafka vs Kinesis — is core backend vocabulary.
⏱ 30 minPrereq: Unit 07You'll be able to: introduce a queue and reason about delivery guarantees
By the end of this unit
Recognize the signals that say "add a queue here."
Compare a message queue (SQS) vs a log/stream (Kafka, Kinesis).
Reason about ordering, at-least-once delivery, and idempotent consumers.
Handle failures: retries, dead-letter queues, backpressure, and size limits.
When to reach for a queue
Add a queue when work can happen later or independently of the user's request. Classic triggers:
Slow work off the hot path: upload a photo → return immediately; transcode/thumbnail in the background.
Spike absorption: a burst of writes queues up and consumers drain at a steady rate instead of crushing the DB.
Fan-out: one event ("post created") triggers many independent reactions (update feeds, notify, index).
Decoupling: producer and consumer can deploy, scale, and fail independently.
The queue buffers between a fast producer and slower/scalable consumers, smoothing spikes and decoupling failure.
Queue vs log: SQS vs Kafka/Kinesis
Task queue (SQS, RabbitMQ)
Message consumed once, then deleted.
Great for work distribution: "process this job."
Easy scaling, managed, simple semantics.
Weaker/limited ordering; not a replayable history.
vs
Log/stream (Kafka, Kinesis)
Append-only log; messages retained and replayable.
Ordered within a partition; multiple independent consumer groups.
Huge throughput; backbone for event-driven systems & analytics.
More to operate; ordering only within a partition.
Pick by intent "If I just need to distribute background jobs, SQS. If I need an ordered, replayable event stream that several teams consume independently — feeds, analytics, CDC — Kafka or Kinesis." Naming replayability and consumer groups shows you know the real difference.
Delivery guarantees & idempotent consumers
Most systems give at-least-once delivery: a message may be delivered more than once (e.g., a consumer crashes after processing but before acking). Exactly-once is expensive and often illusory end-to-end. So the standard pattern is: at-least-once delivery + idempotent consumers — design the handler so processing the same message twice is harmless (dedupe by message id, upsert instead of insert). This echoes idempotency keys from Unit 07.
Ordering: global ordering is costly. Kafka guarantees order only within a partition, so you partition by a key (e.g., user_id) to keep that key's events ordered. Mention this when order matters (a chat, a ledger).
Failure handling
Problem
Mechanism
Transient failure
Retry with exponential backoff + jitter.
Poison message (always fails)
After N retries, move to a dead-letter queue for inspection — don't block the queue.
Respect size limits (SQS ≈ 256 KB). Store the payload in an object store and enqueue a pointer (claim-check pattern).
Watch queue depth A growing backlog is the early-warning sign that consumers are under-provisioned or stuck. It's a key metric to monitor (Unit 13) and a great thing to mention proactively.
Quick check
Users upload videos that take minutes to transcode. How should the upload API behave?
Enqueue and return. Long work must come off the request path or you tie up connections and time out clients. Acknowledge the upload, queue the heavy job, and let a pool of workers transcode and update status — the canonical async pipeline.
Quick check
Your consumers use at-least-once delivery. What must the handler be to stay correct?
Idempotent. Since duplicates are expected, dedupe by message id or use upserts so a redelivery doesn't double-apply an effect (e.g., don't credit an account twice). This is the practical answer to "exactly-once."
Practice before you move on
Insert a queue into the photo app's "post created" flow. List the consumers it fans out to, choose SQS vs Kafka with a reason, describe your retry + dead-letter strategy, and explain how a 1.5 MB image avoids the message size limit.