Exercise 14 · MortgageDisbursementRequested → MortgageDisbursed

Disbursement: offline service and backlog

  • service offline
  • backlog
  • timeout
  • double payout

Principle

When a subscriber is offline, the broker holds the messages. Once back, the service processes the backlog and nothing is lost. The sender does not know about the outage and carries on.

Watch out for the trap: the Process Manager has a timeout and resends the request after 4 s. Two payout requests for the same mortgage then wait in the queue.

Without an idempotent receiver, the bank pays out twice once the service is back. Retries and timeouts therefore always assume idempotency.

What to watch

  • Backlog at Disbursement (badge on the node).
  • Once the service is back (online button), both requests are processed.
  • Number of payouts in the service's local data.

Without the solution / With the solution

Without the solution

Run outage (offline 12 s) + timeout

Disbursement is offline. After a timeout the saga resends the request. In 12 s the service comes back and processes the backlog.

Run in the simulator without the solution

With the solution

Run with idempotent receiver

A second request for the same application returns the original result and the money is paid out once.

Run in the simulator with the solution

What to do: Click “Run outage (offline 12 s) + timeout”. What to look at: The client gets paid out twice. The simulator needs JavaScript and runs as a separate application.

Try it in your own project

Copy the prompt into your coding agent (Claude Code, Copilot, Codex…) and run it in your project's repository.

Ensure that a disbursement service outage (not running, or responding late) neither loses a request nor causes a double payout once it is back.

Language
Variant
# Task for a coding agent: Disbursement: offline service and backlog

Stack: Java. Mode: guided.
Goal: Ensure that a disbursement service outage (not running, or responding late) neither loses a request nor causes a double payout once it is back.

## Context of the existing code

You are working in an existing project (Java backend (typically Java 21, Spring Boot, JPA or JDBC). Use the build tool and style the project already has (Maven/Gradle, JUnit 5).).
Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout).
Prerequisite: the project has an asynchronous application flow up to the contract. If the payout is only a synchronous call, first set up a minimal step: contract message (`LoanContractGenerated`) → disbursement consumer → service → `DisbursementScheduled`.

Read first, change nothing. Find and briefly summarise:
- How messages for the disbursement service are persisted (broker, queue), where the timeouts are and how long an outage the retry policy survives.
- Whether the payout is protected by a business key (applicationId) or an idempotency key.
- Monitoring: who finds out about a growing queue.
- Which of these parts already exist in the project and which are missing. Before writing your own infrastructure (retry, rate limiting, queue monitoring), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan.

## Expected behaviour

An offline disbursement service, once back, processes the backlog the broker held. The trap: after a timeout the sender resends the request; without an idempotent receiver the payout happens twice once the service is back. So every attempt carries the same idempotency key. Example: an attempt times out, yet the service records it; the retry with the same key gets the original payout (`200`), not a second one.

## Specific task

Technical specifics (Java): Unique index on `(application_id, step)`, listener with limited concurrency.

Proceed step by step. After point 1 (plan), write what you found and what you propose, and wait for confirmation:
1. Plan: draw what happens during a 30 s outage (queue, timeout, retries).
2. Introduce an idempotency key `<applicationId>:<process step>` (e.g. `…:disbursement`) with a unique constraint in the disbursement service, in one transaction with the payout. Calls without a key stay unchanged.
3. One policy in one place (attempt timeout, backoff, retry window, rate limit, alarm threshold), shortened in tests. Log a failed attempt as an operational event (failure kind, next attempt or DLQ).
4. Alarm on the age of the oldest message in the queue: threshold longer than the attempt with the longest backoff, shorter than the time to DLQ.
5. Integration test: offline → a request the service processes only after the client timeout → online → exactly one payout. Control: without the key, two payouts.

Watch out for:
- A timeout shorter than the usual processing time produces duplicates on its own.
- When the service comes back, it processes the whole backlog at once: limit the rate, retries included.
- An in-memory broker does not protect against a consumer outage; state this in the conclusion.

## What you must not change

- The existing public API and the behaviour of the calculator endpoints that currently work.
- The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not).
- Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary.
- Public contracts (APIs, event schemas) without backward compatibility.

## Done when

- Integration test with an outage and a late-processed request; control test without the key.
- Unique constraint on the idempotency key in the disbursement service.
- Alarm on queue age and a rate limit.
- JUnit 5 tests in the existing `src/test/java` structure pass with the project's build tool.
- In the summary, state: what you changed, what you assumed, what remains.
# Task for a coding agent: Disbursement: offline service and backlog

Stack: Java. Mode: challenge.
Goal: Ensure that a disbursement service outage (not running, or responding late) neither loses a request nor causes a double payout once it is back.

## Context of the existing code

You are working in an existing project (Java backend (typically Java 21, Spring Boot, JPA or JDBC). Use the build tool and style the project already has (Maven/Gradle, JUnit 5).).
Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout).
Prerequisite: the project has an asynchronous application flow up to the contract. If the payout is only a synchronous call, first set up a minimal step: contract message (`LoanContractGenerated`) → disbursement consumer → service → `DisbursementScheduled`.

Read first, change nothing. Find and briefly summarise:
- How messages for the disbursement service are persisted (broker, queue), where the timeouts are and how long an outage the retry policy survives.
- Whether the payout is protected by a business key (applicationId) or an idempotency key.
- Monitoring: who finds out about a growing queue.
- Which of these parts already exist in the project and which are missing. Before writing your own infrastructure (retry, rate limiting, queue monitoring), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan.

## Expected behaviour

An offline disbursement service, once back, processes the backlog the broker held. The trap: after a timeout the sender resends the request; without an idempotent receiver the payout happens twice once the service is back. So every attempt carries the same idempotency key. Example: an attempt times out, yet the service records it; the retry with the same key gets the original payout (`200`), not a second one.

## Brief (Challenge)

Prove with a test that after a disbursement service outage and a retried request, exactly one payout is made.

Technical specifics (Java): Unique index on `(application_id, step)`, listener with limited concurrency.

Choose your own approach. The pitfalls in this topic are not listed: find them (concurrency, redelivery, ordering, failure midway) and for each write a test that would expose it. At the end, list which pitfalls you found and which you deliberately left open.

## What you must not change

- The existing public API and the behaviour of the calculator endpoints that currently work.
- The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not).
- Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary.
- Public contracts (APIs, event schemas) without backward compatibility.

## Done when

- Integration test with an outage and a late-processed request; control test without the key.
- Unique constraint on the idempotency key in the disbursement service.
- Alarm on queue age and a rate limit.
- JUnit 5 tests in the existing `src/test/java` structure pass with the project's build tool.
- In the summary, state: what you changed, what you assumed, what remains.
# Task for a coding agent: Disbursement: offline service and backlog

Stack: C#. Mode: guided.
Goal: Ensure that a disbursement service outage (not running, or responding late) neither loses a request nor causes a double payout once it is back.

## Context of the existing code

You are working in an existing project (C# backend (.NET, ASP.NET Core, EF Core or Dapper). Use the solution, style and DI conventions the project already has (xUnit/NUnit).).
Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout).
Prerequisite: the project has an asynchronous application flow up to the contract. If the payout is only a synchronous call, first set up a minimal step: contract message (`LoanContractGenerated`) → disbursement consumer → service → `DisbursementScheduled`.

Read first, change nothing. Find and briefly summarise:
- How messages for the disbursement service are persisted (broker, queue), where the timeouts are and how long an outage the retry policy survives.
- Whether the payout is protected by a business key (applicationId) or an idempotency key.
- Monitoring: who finds out about a growing queue.
- Which of these parts already exist in the project and which are missing. Before writing your own infrastructure (retry, rate limiting, queue monitoring), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan.

## Expected behaviour

An offline disbursement service, once back, processes the backlog the broker held. The trap: after a timeout the sender resends the request; without an idempotent receiver the payout happens twice once the service is back. So every attempt carries the same idempotency key. Example: an attempt times out, yet the service records it; the retry with the same key gets the original payout (`200`), not a second one.

## Specific task

Technical specifics (C#): Unique index, consumer with `PrefetchCount`/`ConcurrentMessageLimit`.

Proceed step by step. After point 1 (plan), write what you found and what you propose, and wait for confirmation:
1. Plan: draw what happens during a 30 s outage (queue, timeout, retries).
2. Introduce an idempotency key `<applicationId>:<process step>` (e.g. `…:disbursement`) with a unique constraint in the disbursement service, in one transaction with the payout. Calls without a key stay unchanged.
3. One policy in one place (attempt timeout, backoff, retry window, rate limit, alarm threshold), shortened in tests. Log a failed attempt as an operational event (failure kind, next attempt or DLQ).
4. Alarm on the age of the oldest message in the queue: threshold longer than the attempt with the longest backoff, shorter than the time to DLQ.
5. Integration test: offline → a request the service processes only after the client timeout → online → exactly one payout. Control: without the key, two payouts.

Watch out for:
- A timeout shorter than the usual processing time produces duplicates on its own.
- When the service comes back, it processes the whole backlog at once: limit the rate, retries included.
- An in-memory broker does not protect against a consumer outage; state this in the conclusion.

## What you must not change

- The existing public API and the behaviour of the calculator endpoints that currently work.
- The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not).
- Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary.
- Public contracts (APIs, event schemas) without backward compatibility.

## Done when

- Integration test with an outage and a late-processed request; control test without the key.
- Unique constraint on the idempotency key in the disbursement service.
- Alarm on queue age and a rate limit.
- Tests in the existing test project pass (`dotnet test`).
- In the summary, state: what you changed, what you assumed, what remains.
# Task for a coding agent: Disbursement: offline service and backlog

Stack: C#. Mode: challenge.
Goal: Ensure that a disbursement service outage (not running, or responding late) neither loses a request nor causes a double payout once it is back.

## Context of the existing code

You are working in an existing project (C# backend (.NET, ASP.NET Core, EF Core or Dapper). Use the solution, style and DI conventions the project already has (xUnit/NUnit).).
Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout).
Prerequisite: the project has an asynchronous application flow up to the contract. If the payout is only a synchronous call, first set up a minimal step: contract message (`LoanContractGenerated`) → disbursement consumer → service → `DisbursementScheduled`.

Read first, change nothing. Find and briefly summarise:
- How messages for the disbursement service are persisted (broker, queue), where the timeouts are and how long an outage the retry policy survives.
- Whether the payout is protected by a business key (applicationId) or an idempotency key.
- Monitoring: who finds out about a growing queue.
- Which of these parts already exist in the project and which are missing. Before writing your own infrastructure (retry, rate limiting, queue monitoring), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan.

## Expected behaviour

An offline disbursement service, once back, processes the backlog the broker held. The trap: after a timeout the sender resends the request; without an idempotent receiver the payout happens twice once the service is back. So every attempt carries the same idempotency key. Example: an attempt times out, yet the service records it; the retry with the same key gets the original payout (`200`), not a second one.

## Brief (Challenge)

Prove with a test that after a disbursement service outage and a retried request, exactly one payout is made.

Technical specifics (C#): Unique index, consumer with `PrefetchCount`/`ConcurrentMessageLimit`.

Choose your own approach. The pitfalls in this topic are not listed: find them (concurrency, redelivery, ordering, failure midway) and for each write a test that would expose it. At the end, list which pitfalls you found and which you deliberately left open.

## What you must not change

- The existing public API and the behaviour of the calculator endpoints that currently work.
- The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not).
- Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary.
- Public contracts (APIs, event schemas) without backward compatibility.

## Done when

- Integration test with an outage and a late-processed request; control test without the key.
- Unique constraint on the idempotency key in the disbursement service.
- Alarm on queue age and a rate limit.
- Tests in the existing test project pass (`dotnet test`).
- In the summary, state: what you changed, what you assumed, what remains.
# Task for a coding agent: Disbursement: offline service and backlog

Stack: Web (HTML/JS or the app's stack). Mode: guided.
Goal: Ensure that a disbursement service outage (not running, or responding late) neither loses a request nor causes a double payout once it is back.

## Context of the existing code

You are working in an existing project (web application: a browser frontend, possibly with its own backend. If the project has a backend, the event store, broker and services belong on the backend, and the points about endpoints and tables refer to it. If it has no backend, use IndexedDB in the browser and `fake-indexeddb` in tests. Keep the project's framework and test runner; add new dependencies only with a justification.).
Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout).
Prerequisite: the project has an asynchronous application flow up to the contract. If the payout is only a synchronous call, first set up a minimal step: contract message (`LoanContractGenerated`) → disbursement consumer → service → `DisbursementScheduled`.

Read first, change nothing. Find and briefly summarise:
- How messages for the disbursement service are persisted (broker, queue), where the timeouts are and how long an outage the retry policy survives.
- Whether the payout is protected by a business key (applicationId) or an idempotency key.
- Monitoring: who finds out about a growing queue.
- Which of these parts already exist in the project and which are missing. Before writing your own infrastructure (retry, rate limiting, queue monitoring), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan.

## Expected behaviour

An offline disbursement service, once back, processes the backlog the broker held. The trap: after a timeout the sender resends the request; without an idempotent receiver the payout happens twice once the service is back. So every attempt carries the same idempotency key. Example: an attempt times out, yet the service records it; the retry with the same key gets the original payout (`200`), not a second one.

## Specific task

Technical specifics (Web (HTML/JS or the app's stack)): With a backend, the disbursement consumer runs on the server and the UI only reads the “payout pending” state; without a backend, a queue in IndexedDB. Never retry a command without an idempotency key.

Proceed step by step. After point 1 (plan), write what you found and what you propose, and wait for confirmation:
1. Plan: draw what happens during a 30 s outage (queue, timeout, retries).
2. Introduce an idempotency key `<applicationId>:<process step>` (e.g. `…:disbursement`) with a unique constraint in the disbursement service, in one transaction with the payout. Calls without a key stay unchanged.
3. One policy in one place (attempt timeout, backoff, retry window, rate limit, alarm threshold), shortened in tests. Log a failed attempt as an operational event (failure kind, next attempt or DLQ).
4. Alarm on the age of the oldest message in the queue: threshold longer than the attempt with the longest backoff, shorter than the time to DLQ.
5. Integration test: offline → a request the service processes only after the client timeout → online → exactly one payout. Control: without the key, two payouts.

Watch out for:
- A timeout shorter than the usual processing time produces duplicates on its own.
- When the service comes back, it processes the whole backlog at once: limit the rate, retries included.
- An in-memory broker does not protect against a consumer outage; state this in the conclusion.

## What you must not change

- The existing public API and the behaviour of the calculator endpoints that currently work.
- The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not).
- Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary.
- Public contracts (APIs, event schemas) without backward compatibility.

## Done when

- Integration test with an outage and a late-processed request; control test without the key.
- Unique constraint on the idempotency key in the disbursement service.
- Alarm on queue age and a rate limit.
- Tests pass with the existing test runner; lint and build pass if the project has them.
- In the summary, state: what you changed, what you assumed, what remains.
# Task for a coding agent: Disbursement: offline service and backlog

Stack: Web (HTML/JS or the app's stack). Mode: challenge.
Goal: Ensure that a disbursement service outage (not running, or responding late) neither loses a request nor causes a double payout once it is back.

## Context of the existing code

You are working in an existing project (web application: a browser frontend, possibly with its own backend. If the project has a backend, the event store, broker and services belong on the backend, and the points about endpoints and tables refer to it. If it has no backend, use IndexedDB in the browser and `fake-indexeddb` in tests. Keep the project's framework and test runner; add new dependencies only with a justification.).
Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout).
Prerequisite: the project has an asynchronous application flow up to the contract. If the payout is only a synchronous call, first set up a minimal step: contract message (`LoanContractGenerated`) → disbursement consumer → service → `DisbursementScheduled`.

Read first, change nothing. Find and briefly summarise:
- How messages for the disbursement service are persisted (broker, queue), where the timeouts are and how long an outage the retry policy survives.
- Whether the payout is protected by a business key (applicationId) or an idempotency key.
- Monitoring: who finds out about a growing queue.
- Which of these parts already exist in the project and which are missing. Before writing your own infrastructure (retry, rate limiting, queue monitoring), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan.

## Expected behaviour

An offline disbursement service, once back, processes the backlog the broker held. The trap: after a timeout the sender resends the request; without an idempotent receiver the payout happens twice once the service is back. So every attempt carries the same idempotency key. Example: an attempt times out, yet the service records it; the retry with the same key gets the original payout (`200`), not a second one.

## Brief (Challenge)

Prove with a test that after a disbursement service outage and a retried request, exactly one payout is made.

Technical specifics (Web (HTML/JS or the app's stack)): With a backend, the disbursement consumer runs on the server and the UI only reads the “payout pending” state; without a backend, a queue in IndexedDB. Never retry a command without an idempotency key.

Choose your own approach. The pitfalls in this topic are not listed: find them (concurrency, redelivery, ordering, failure midway) and for each write a test that would expose it. At the end, list which pitfalls you found and which you deliberately left open.

## What you must not change

- The existing public API and the behaviour of the calculator endpoints that currently work.
- The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not).
- Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary.
- Public contracts (APIs, event schemas) without backward compatibility.

## Done when

- Integration test with an outage and a late-processed request; control test without the key.
- Unique constraint on the idempotency key in the disbursement service.
- Alarm on queue age and a rate limit.
- Tests pass with the existing test runner; lint and build pass if the project has them.
- In the summary, state: what you changed, what you assumed, what remains.

Training for your team

Would you like this training for your team?

We run the training on your code and your domain problem. We agree the scope, dates and format with you.

Discuss the training