Exercise 11 · CustomerIdentityCreationRequested → CustomerIdentityCreated

Customer Identity: retry and DLQ

retry
backoff
DLQ
poison message
Process Manager

Principle

After the mortgage is approved, the Process Manager asks Customer Identity Service with the CustomerIdentityCreationRequested command. The service replies with CustomerIdentityCreated.

The service may fail temporarily. Without retries the message is lost and the process hangs. Retry with exponential backoff gives the service time to recover.

Once the attempts are exhausted, the message ends up in the Dead Letter Queue, where nobody forgets about it. A poison message (always fails) must not be retried forever. After the cause is fixed, operations return it with the redrive button.

What to watch

The number of attempts and the growing pauses (backoff) on the message.
The service “recovers” within 4 s: with retry the process completes, without it, it doesn't.
The “Poison the next message” button and the subsequent redrive from the DLQ.

Without the solution

Run without retry

Identity Service returns an error. It recovers within 4 s, but the message has already been discarded and the Process Manager waits forever.

Run in the simulator without the solution

With the solution

Run with retry and DLQ

The message is retried with pauses of 0.6 / 1.2 / 2.4 s and goes through once the service recovers. Also try a poison message: it ends up in the DLQ.

Run in the simulator with the solution

What to do: Click “Run without retry”. What to look at: The message is lost after the first error and the Process Manager waits forever. The simulator needs JavaScript and runs as a separate application.

Try it in your own project

Copy the prompt into your coding agent (Claude Code, Copilot, Codex…) and run it in your project's repository.

Add retry with exponential backoff and a DLQ for the event consumer, and be able to replay a message from the DLQ.

LanguageJavaC#Web (HTML/JS or the app's stack)

VariantGuided promptStep by step, with warnings about pitfalls and pauses for confirmation.ChallengeOnly the goal and acceptance. The agent finds the pitfalls itself and lists them at the end.

# Task for a coding agent: Customer Identity: retry and DLQ

Stack: Java. Mode: guided.
Goal: Add retry with exponential backoff and a DLQ for the event consumer, and be able to replay a message from the DLQ.

## Context of the existing code

You are working in an existing project (Java backend (typically Java 21, Spring Boot, JPA or JDBC). Use the build tool and style the project already has (Maven/Gradle, JUnit 5).).
Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout).
Prerequisite: the project has a broker with consumers and the `MortgageApproved` event. If not, first create a minimal in-memory broker and this event.

Read first, change nothing. Find and briefly summarise:
- The Customer Identity message consumer (subscribes to `MortgageApproved`, verifies identity with the identity service) and how it handles errors today (ack/nack). If it doesn't exist, design it in the plan.
- Broker configuration: retry, DLQ, max attempts.
- Existing shared retry libraries (e.g. `p-retry`, `cockatiel`; in .NET Polly) in the project/organisation.
- Which of these parts already exist in the project and which are missing. Before you write your own infrastructure (broker client, outbox, retry), check whether the project or organisation already has it as a ready-made library, and use it.

## Expected behaviour

The identity service may fail temporarily; without retries the message is lost and the process hangs. Retry with backoff (pauses of 0.6 / 1.2 / 2.4 s) gives the service time: if it recovers within 4 s, the message goes through. A poison message (always fails) ends up in the DLQ and operations return it via redrive after the fix. Example: a 503 response once → the 2nd attempt goes through; 422 → straight to the DLQ.

## Specific task

Technical specifics (Java): Spring Retry/Resilience4j or the broker's native retry+DLQ (Kafka `DefaultErrorHandler` + `DeadLetterPublishingRecoverer`).

Proceed step by step. After point 1 (plan), write what you found and what you propose, and wait for confirmation:
1. Plan: which errors are transient (retry) and which are permanent (straight to the DLQ).
2. Use the project's/broker's existing retry mechanism; don't write a loop in the handler. If there is none, add delayed redelivery of the message to the broker consumer and state it.
3. Backoff with jitter and an attempt limit, after which the DLQ with headers: reason, error text, attempt count, consumer.
4. An operational redrive tool (command or endpoint): delivers the message only to the consumer it failed for, not back to the topic. Protection against an infinite loop: a message cannot be returned twice, a redrive limit.
5. Tests (fake service failing on command, fake timer): a transient error goes through on the 2nd attempt; a permanent error ends up in the DLQ; redrive after the fix goes through.

Watch out for:
- Retry without idempotency creates duplicates (solved by an idempotent receiver with an Inbox).
- Retry storm: everyone retries at once and, without jitter, overwhelms the service.

## What you must not change

- The existing public API and the behaviour of the calculator endpoints that currently work.
- The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not).
- Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary.
- Public contracts (APIs, event schemas) without backward compatibility.

## Done when

- Three tests: transient error, permanent error, redrive.
- The DLQ carries the reason and the attempt count.
- There is no custom retry loop in the consumer's handler (command retries are out of scope).
- JUnit 5 tests in the existing `src/test/java` structure pass with the project's build tool.
- In the summary, state: what you changed, what you assumed, what remains.

# Task for a coding agent: Customer Identity: retry and DLQ

Stack: Java. Mode: challenge.
Goal: Add retry with exponential backoff and a DLQ for the event consumer, and be able to replay a message from the DLQ.

## Context of the existing code

## Expected behaviour

## Brief (Challenge)

Ensure that a transient failure of the CustomerIdentityCreationRequested consumer doesn't lose the message and that a permanent failure ends up in the DLQ with the option of a redrive.

Technical specifics (Java): Spring Retry/Resilience4j or the broker's native retry+DLQ (Kafka `DefaultErrorHandler` + `DeadLetterPublishingRecoverer`).

Choose your own approach. The pitfalls in this topic are not listed: find them (concurrency, redelivery, ordering, failure midway) and for each write a test that would expose it. At the end, list which pitfalls you found and which you deliberately left open.

## What you must not change

## Done when

# Task for a coding agent: Customer Identity: retry and DLQ

Stack: C#. Mode: guided.
Goal: Add retry with exponential backoff and a DLQ for the event consumer, and be able to replay a message from the DLQ.

## Context of the existing code

You are working in an existing project (C# backend (.NET, ASP.NET Core, EF Core or Dapper). Use the solution, style and DI conventions the project already has (xUnit/NUnit).).
Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout).
Prerequisite: the project has a broker with consumers and the `MortgageApproved` event. If not, first create a minimal in-memory broker and this event.

## Expected behaviour

## Specific task

Technical specifics (C#): The retry/DLQ layer of the project's messaging library (MassTransit/NServiceBus/Service Bus) or Polly.

Watch out for:
- Retry without idempotency creates duplicates (solved by an idempotent receiver with an Inbox).
- Retry storm: everyone retries at once and, without jitter, overwhelms the service.

## What you must not change

## Done when

- Three tests: transient error, permanent error, redrive.
- The DLQ carries the reason and the attempt count.
- There is no custom retry loop in the consumer's handler (command retries are out of scope).
- Tests in the existing test project pass (`dotnet test`).
- In the summary, state: what you changed, what you assumed, what remains.

# Task for a coding agent: Customer Identity: retry and DLQ

Stack: C#. Mode: challenge.
Goal: Add retry with exponential backoff and a DLQ for the event consumer, and be able to replay a message from the DLQ.

## Context of the existing code

## Expected behaviour

## Brief (Challenge)

Ensure that a transient failure of the CustomerIdentityCreationRequested consumer doesn't lose the message and that a permanent failure ends up in the DLQ with the option of a redrive.

Technical specifics (C#): The retry/DLQ layer of the project's messaging library (MassTransit/NServiceBus/Service Bus) or Polly.

## What you must not change

## Done when

- Three tests: transient error, permanent error, redrive.
- The DLQ carries the reason and the attempt count.
- There is no custom retry loop in the consumer's handler (command retries are out of scope).
- Tests in the existing test project pass (`dotnet test`).
- In the summary, state: what you changed, what you assumed, what remains.

# Task for a coding agent: Customer Identity: retry and DLQ

Stack: Web (HTML/JS or the app's stack). Mode: guided.
Goal: Add retry with exponential backoff and a DLQ for the event consumer, and be able to replay a message from the DLQ.

## Context of the existing code

You are working in an existing project (web application: a browser frontend, possibly with its own backend. If the project has a backend, the event store, broker and services belong on the backend, and the points about endpoints and tables refer to it. If it has no backend, use IndexedDB in the browser and `fake-indexeddb` in tests. Keep the project's framework and test runner; add new dependencies only with a justification.).
Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout).
Prerequisite: the project has a broker with consumers and the `MortgageApproved` event. If not, first create a minimal in-memory broker and this event.

## Expected behaviour

## Specific task

Technical specifics (Web (HTML/JS or the app's stack)): With a backend, use its broker and DLQ; without a backend, retries with backoff and a failure queue in IndexedDB.

Watch out for:
- Retry without idempotency creates duplicates (solved by an idempotent receiver with an Inbox).
- Retry storm: everyone retries at once and, without jitter, overwhelms the service.

## What you must not change

## Done when

- Three tests: transient error, permanent error, redrive.
- The DLQ carries the reason and the attempt count.
- There is no custom retry loop in the consumer's handler (command retries are out of scope).
- Tests pass with the existing test runner; lint and build pass if the project has them.
- In the summary, state: what you changed, what you assumed, what remains.

# Task for a coding agent: Customer Identity: retry and DLQ

Stack: Web (HTML/JS or the app's stack). Mode: challenge.
Goal: Add retry with exponential backoff and a DLQ for the event consumer, and be able to replay a message from the DLQ.

## Context of the existing code

## Expected behaviour

## Brief (Challenge)

Ensure that a transient failure of the CustomerIdentityCreationRequested consumer doesn't lose the message and that a permanent failure ends up in the DLQ with the option of a redrive.

Technical specifics (Web (HTML/JS or the app's stack)): With a backend, use its broker and DLQ; without a backend, retries with backoff and a failure queue in IndexedDB.

## What you must not change

## Done when

- Three tests: transient error, permanent error, redrive.
- The DLQ carries the reason and the attempt count.
- There is no custom retry loop in the consumer's handler (command retries are out of scope).
- Tests pass with the existing test runner; lint and build pass if the project has them.
- In the summary, state: what you changed, what you assumed, what remains.

Training for your team

Would you like this training for your team?

We run the training on your code and your domain problem. We agree the scope, dates and format with you.

Discuss the training

Customer Identity: retry and DLQ

Principle

What to watch

Without the solution / With the solution

Run without retry

Run with retry and DLQ

Try it in your own project

Related parts of the presentation

Would you like this training for your team?