Part 14 of 16 · Day 2 · retry, DLQ, backpressure
The queue grows and nobody knows why
Problem and experience
The queue grows and nobody knows why
A slow service or a peak, and the queue grows. The sender keeps sending. Without metrics we find out from the client.
- Senderrychle
- Queuebacklog grows
- Slow service
- Client is waiting
From practiceHow do you find out that something is piling up before the client calls?
TakeawayWithout observation we discover a problem only when the client sees it.
What it means
Backlog is the number of messages waiting to be processed. Backpressure is a way of telling the sender to slow down.
Why we use it
The sender must not flood the receiver indefinitely.
When it makes sense
For every asynchronous connection.
Where it breaks
- An unbounded queue.
- Processing an old message without checking that it is still valid.
- No metric for the age of the oldest message.
Imagine
Disbursement is offline for an hour. Applications wait in the queue and nobody knows.
What we cannot see, we cannot fix.
Principle, application and exercise
Retry, DLQ, backpressure and observability
We retry with a growing pause, park permanent errors in the DLQ and limit the load. And we see what is happening.
Resilience
- Retrybackoff
- DLQonce exhausted
- Backpressurequeue limit
- Timeoutfor process steps
Observability
- Queue length
- Age of the oldest message
- Count in DLQ
- Tracing by correlationId
Trade-offRetries can overload a service (a retry storm), limits increase latency, and metrics need care and dashboards that someone actually watches.
TakeawayAsynchronous delivery survives an outage. Timeouts, idempotency and metrics guard what is left of it.
What it means
The service is offline, messages wait. When it comes back, it processes them. The Process Manager retries the request after a timeout, and idempotency prevents a double effect.
Why we use it
The sender need not know about the outage. Operations must.
When it makes sense
Whenever the receiver does not have to respond immediately.
Where it breaks
- Retry storms and missing jitter.
- Timeout without idempotency: a double payout.
- Unmonitored backlog and DLQ.
Imagine
Disbursement is offline. Applications wait, the service comes back up, processes the queue and pays out each mortgage once.
A good system survives an outage. The best one does not even notice it.
Training for your team
Would you like this training for your team?
We run the training on your code and your domain problem. We agree the scope, dates and format with you.
Discuss the training →