Exercise 17 · All failures at once and all defences
The whole system under pressure
- souhrn
- chaos
- Outbox + Inbox + Saga
Principle
Finally, all failures combine: lost publication, service errors, duplicates, shuffled order, lost acknowledgements.
The defences complement each other: Outbox guarantees publication, retry+DLQ handles transient errors, Inbox/idempotency handles duplicates, corrSeq handles order and Saga compensates the irreversible.
None of them is enough on its own. Turn them on one at a time and watch which cause each one removes.
What to watch
- The “Cause → Effect” diagnosis on the right.
- Which switch removes which unwanted effects.
Without the solution / With the solution
Without the solution
Run chaos without defences
Services fail, messages are duplicated and lost. No defence is enabled.
Run in the simulator without the solutionWith the solution
Run chaos with all defences
The same conditions, but with defences enabled. The process completes consistently, just more slowly.
Run in the simulator with the solutionWhat to do: Click “Run chaos without defences”. What to look at: The process does not complete consistently: duplicates, a double payout or a hanging process appear. The simulator needs JavaScript and runs as a separate application.
Try it in your own project
Copy the prompt into your coding agent (Claude Code, Copilot, Codex…) and run it in your project's repository.
Build a resilience test (chaos test) of the whole onboarding and use it to determine which defences are still missing.
# Task for a coding agent: The whole system under pressure Stack: Java. Mode: guided. Goal: Build a resilience test (chaos test) of the whole onboarding and use it to determine which defences are still missing. ## Context of the existing code You are working in an existing project (Java backend (typically Java 21, Spring Boot, JPA or JDBC). Use the build tool and style the project already has (Maven/Gradle, JUnit 5).). Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout). Prerequisite: the project has asynchronous onboarding via a broker all the way to payout; the harness will show the missing defences. If the flow is missing, first create a minimal in-memory flow. Read first, change nothing. Find and briefly summarise: - Existing onboarding integration/e2e tests. - Fault-injection options: what takes time, timers, randomness and the network when the system is composed; inject faults and virtual time there as optional parameters (default behaviour unchanged). - Metrics: queues, DLQ, message age. - Which of these parts already exist in the project and which are missing. Before writing your own test infrastructure (virtual time, PRNG, fake broker), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan. ## Expected behaviour With all faults at once, the defences complement each other (Outbox, retry and DLQ, Inbox and idempotency, correlation sequence, Saga) and none is enough on its own; with all of them the process completes consistently, just more slowly. Example: the “duplicate” profile (the document service writes the contract, the response is lost) finds two contracts for one application. ## Specific task Technical specifics (Java): JUnit + Toxiproxy/Testcontainers or a fake broker with injected faults. Proceed step by step. After point 1 (plan), write what you found and what you propose, and wait for confirmation: 1. Plan: a table of failure × defence × invariant. Loss: a consumer stops consuming for a while. Duplicate: delivery 2×, lost publish acknowledgement, lost response after a write. Order: a message reaches a consumer after the next one. Offline: a service, the broker or the browser. Slow service: response after the client timeout, the write goes through. 2. A test harness with injected faults and a fixed seed: profile = one kind of failure, a separate PRNG for each kind of fault and for jitter, and after the run a snapshot of the world (service tables, broker log, read models, process state). 3. Post-run invariants as pure functions of the snapshot (source of truth = service tables and the broker log): a completed application has exactly one account, one contract and one payout; a rejected or reverted one has nothing open; read models = source of truth; nothing is left hanging. Also measure side effects (extra calls, retries, DLQ). 4. Run the harness for each profile with seeds 1–10 and print a table: profile × invariant, first violating seed, missing defence. 5. Fix one gap: the most frequent violation with business impact that can be fixed without changing other people's tests. Put the rest in the backlog. Watch out for: - A test that fails once in a hundred runs is worse than none: a fixed seed and multiple seeds. - Use a temporary mutation of a defence to verify that the harness really finds the violation. - Someone else's test that is sensitive to real time may fail even on the baseline: do not change it, and mention it in the conclusion. ## What you must not change - The existing public API and the behaviour of the calculator endpoints that currently work. - The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not). - Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary. - Public contracts (APIs, event schemas) without backward compatibility. ## Done when - The test is reproducible: two runs with the same seed give the same fingerprint; the seed is in the test name. - Invariants are explicit (in code and in the README). - Open risks are in the README, each with a test; the fixed gap has a test that failed before the fix. - JUnit 5 tests in the existing `src/test/java` structure pass with the project's build tool. - In the summary, state: what you changed, what you assumed, what remains.
# Task for a coding agent: The whole system under pressure Stack: Java. Mode: challenge. Goal: Build a resilience test (chaos test) of the whole onboarding and use it to determine which defences are still missing. ## Context of the existing code You are working in an existing project (Java backend (typically Java 21, Spring Boot, JPA or JDBC). Use the build tool and style the project already has (Maven/Gradle, JUnit 5).). Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout). Prerequisite: the project has asynchronous onboarding via a broker all the way to payout; the harness will show the missing defences. If the flow is missing, first create a minimal in-memory flow. Read first, change nothing. Find and briefly summarise: - Existing onboarding integration/e2e tests. - Fault-injection options: what takes time, timers, randomness and the network when the system is composed; inject faults and virtual time there as optional parameters (default behaviour unchanged). - Metrics: queues, DLQ, message age. - Which of these parts already exist in the project and which are missing. Before writing your own test infrastructure (virtual time, PRNG, fake broker), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan. ## Expected behaviour With all faults at once, the defences complement each other (Outbox, retry and DLQ, Inbox and idempotency, correlation sequence, Saga) and none is enough on its own; with all of them the process completes consistently, just more slowly. Example: the “duplicate” profile (the document service writes the contract, the response is lost) finds two contracts for one application. ## Brief (Challenge) Write a reproducible chaos test of onboarding with post-run invariants and show which failures are not yet handled. Technical specifics (Java): JUnit + Toxiproxy/Testcontainers or a fake broker with injected faults. Choose your own approach. The pitfalls in this topic are not listed: find them (concurrency, redelivery, ordering, failure midway) and for each write a test that would expose it. At the end, list which pitfalls you found and which you deliberately left open. ## What you must not change - The existing public API and the behaviour of the calculator endpoints that currently work. - The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not). - Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary. - Public contracts (APIs, event schemas) without backward compatibility. ## Done when - The test is reproducible: two runs with the same seed give the same fingerprint; the seed is in the test name. - Invariants are explicit (in code and in the README). - Open risks are in the README, each with a test; the fixed gap has a test that failed before the fix. - JUnit 5 tests in the existing `src/test/java` structure pass with the project's build tool. - In the summary, state: what you changed, what you assumed, what remains.
# Task for a coding agent: The whole system under pressure Stack: C#. Mode: guided. Goal: Build a resilience test (chaos test) of the whole onboarding and use it to determine which defences are still missing. ## Context of the existing code You are working in an existing project (C# backend (.NET, ASP.NET Core, EF Core or Dapper). Use the solution, style and DI conventions the project already has (xUnit/NUnit).). Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout). Prerequisite: the project has asynchronous onboarding via a broker all the way to payout; the harness will show the missing defences. If the flow is missing, first create a minimal in-memory flow. Read first, change nothing. Find and briefly summarise: - Existing onboarding integration/e2e tests. - Fault-injection options: what takes time, timers, randomness and the network when the system is composed; inject faults and virtual time there as optional parameters (default behaviour unchanged). - Metrics: queues, DLQ, message age. - Which of these parts already exist in the project and which are missing. Before writing your own test infrastructure (virtual time, PRNG, fake broker), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan. ## Expected behaviour With all faults at once, the defences complement each other (Outbox, retry and DLQ, Inbox and idempotency, correlation sequence, Saga) and none is enough on its own; with all of them the process completes consistently, just more slowly. Example: the “duplicate” profile (the document service writes the contract, the response is lost) finds two contracts for one application. ## Specific task Technical specifics (C#): xUnit + a broker test harness (e.g. MassTransit test harness) with injected faults. Proceed step by step. After point 1 (plan), write what you found and what you propose, and wait for confirmation: 1. Plan: a table of failure × defence × invariant. Loss: a consumer stops consuming for a while. Duplicate: delivery 2×, lost publish acknowledgement, lost response after a write. Order: a message reaches a consumer after the next one. Offline: a service, the broker or the browser. Slow service: response after the client timeout, the write goes through. 2. A test harness with injected faults and a fixed seed: profile = one kind of failure, a separate PRNG for each kind of fault and for jitter, and after the run a snapshot of the world (service tables, broker log, read models, process state). 3. Post-run invariants as pure functions of the snapshot (source of truth = service tables and the broker log): a completed application has exactly one account, one contract and one payout; a rejected or reverted one has nothing open; read models = source of truth; nothing is left hanging. Also measure side effects (extra calls, retries, DLQ). 4. Run the harness for each profile with seeds 1–10 and print a table: profile × invariant, first violating seed, missing defence. 5. Fix one gap: the most frequent violation with business impact that can be fixed without changing other people's tests. Put the rest in the backlog. Watch out for: - A test that fails once in a hundred runs is worse than none: a fixed seed and multiple seeds. - Use a temporary mutation of a defence to verify that the harness really finds the violation. - Someone else's test that is sensitive to real time may fail even on the baseline: do not change it, and mention it in the conclusion. ## What you must not change - The existing public API and the behaviour of the calculator endpoints that currently work. - The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not). - Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary. - Public contracts (APIs, event schemas) without backward compatibility. ## Done when - The test is reproducible: two runs with the same seed give the same fingerprint; the seed is in the test name. - Invariants are explicit (in code and in the README). - Open risks are in the README, each with a test; the fixed gap has a test that failed before the fix. - Tests in the existing test project pass (`dotnet test`). - In the summary, state: what you changed, what you assumed, what remains.
# Task for a coding agent: The whole system under pressure Stack: C#. Mode: challenge. Goal: Build a resilience test (chaos test) of the whole onboarding and use it to determine which defences are still missing. ## Context of the existing code You are working in an existing project (C# backend (.NET, ASP.NET Core, EF Core or Dapper). Use the solution, style and DI conventions the project already has (xUnit/NUnit).). Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout). Prerequisite: the project has asynchronous onboarding via a broker all the way to payout; the harness will show the missing defences. If the flow is missing, first create a minimal in-memory flow. Read first, change nothing. Find and briefly summarise: - Existing onboarding integration/e2e tests. - Fault-injection options: what takes time, timers, randomness and the network when the system is composed; inject faults and virtual time there as optional parameters (default behaviour unchanged). - Metrics: queues, DLQ, message age. - Which of these parts already exist in the project and which are missing. Before writing your own test infrastructure (virtual time, PRNG, fake broker), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan. ## Expected behaviour With all faults at once, the defences complement each other (Outbox, retry and DLQ, Inbox and idempotency, correlation sequence, Saga) and none is enough on its own; with all of them the process completes consistently, just more slowly. Example: the “duplicate” profile (the document service writes the contract, the response is lost) finds two contracts for one application. ## Brief (Challenge) Write a reproducible chaos test of onboarding with post-run invariants and show which failures are not yet handled. Technical specifics (C#): xUnit + a broker test harness (e.g. MassTransit test harness) with injected faults. Choose your own approach. The pitfalls in this topic are not listed: find them (concurrency, redelivery, ordering, failure midway) and for each write a test that would expose it. At the end, list which pitfalls you found and which you deliberately left open. ## What you must not change - The existing public API and the behaviour of the calculator endpoints that currently work. - The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not). - Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary. - Public contracts (APIs, event schemas) without backward compatibility. ## Done when - The test is reproducible: two runs with the same seed give the same fingerprint; the seed is in the test name. - Invariants are explicit (in code and in the README). - Open risks are in the README, each with a test; the fixed gap has a test that failed before the fix. - Tests in the existing test project pass (`dotnet test`). - In the summary, state: what you changed, what you assumed, what remains.
# Task for a coding agent: The whole system under pressure Stack: Web (HTML/JS or the app's stack). Mode: guided. Goal: Build a resilience test (chaos test) of the whole onboarding and use it to determine which defences are still missing. ## Context of the existing code You are working in an existing project (web application: a browser frontend, possibly with its own backend. If the project has a backend, the event store, broker and services belong on the backend, and the points about endpoints and tables refer to it. If it has no backend, use IndexedDB in the browser and `fake-indexeddb` in tests. Keep the project's framework and test runner; add new dependencies only with a justification.). Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout). Prerequisite: the project has asynchronous onboarding via a broker all the way to payout; the harness will show the missing defences. If the flow is missing, first create a minimal in-memory flow. Read first, change nothing. Find and briefly summarise: - Existing onboarding integration/e2e tests. - Fault-injection options: what takes time, timers, randomness and the network when the system is composed; inject faults and virtual time there as optional parameters (default behaviour unchanged). - Metrics: queues, DLQ, message age. - Which of these parts already exist in the project and which are missing. Before writing your own test infrastructure (virtual time, PRNG, fake broker), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan. ## Expected behaviour With all faults at once, the defences complement each other (Outbox, retry and DLQ, Inbox and idempotency, correlation sequence, Saga) and none is enough on its own; with all of them the process completes consistently, just more slowly. Example: the “duplicate” profile (the document service writes the contract, the response is lost) finds two contracts for one application. ## Specific task Technical specifics (Web (HTML/JS or the app's stack)): Vitest with a deterministic PRNG (seed) and fake time: only the harness advances the clock and timers. The whole run is in a single process without a network (API, service HTTP handlers, in-memory broker); fake timers over real HTTP are not reproducible. Proceed step by step. After point 1 (plan), write what you found and what you propose, and wait for confirmation: 1. Plan: a table of failure × defence × invariant. Loss: a consumer stops consuming for a while. Duplicate: delivery 2×, lost publish acknowledgement, lost response after a write. Order: a message reaches a consumer after the next one. Offline: a service, the broker or the browser. Slow service: response after the client timeout, the write goes through. 2. A test harness with injected faults and a fixed seed: profile = one kind of failure, a separate PRNG for each kind of fault and for jitter, and after the run a snapshot of the world (service tables, broker log, read models, process state). 3. Post-run invariants as pure functions of the snapshot (source of truth = service tables and the broker log): a completed application has exactly one account, one contract and one payout; a rejected or reverted one has nothing open; read models = source of truth; nothing is left hanging. Also measure side effects (extra calls, retries, DLQ). 4. Run the harness for each profile with seeds 1–10 and print a table: profile × invariant, first violating seed, missing defence. 5. Fix one gap: the most frequent violation with business impact that can be fixed without changing other people's tests. Put the rest in the backlog. Watch out for: - A test that fails once in a hundred runs is worse than none: a fixed seed and multiple seeds. - Use a temporary mutation of a defence to verify that the harness really finds the violation. - Someone else's test that is sensitive to real time may fail even on the baseline: do not change it, and mention it in the conclusion. ## What you must not change - The existing public API and the behaviour of the calculator endpoints that currently work. - The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not). - Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary. - Public contracts (APIs, event schemas) without backward compatibility. ## Done when - The test is reproducible: two runs with the same seed give the same fingerprint; the seed is in the test name. - Invariants are explicit (in code and in the README). - Open risks are in the README, each with a test; the fixed gap has a test that failed before the fix. - Tests pass with the existing test runner; lint and build pass if the project has them. - In the summary, state: what you changed, what you assumed, what remains.
# Task for a coding agent: The whole system under pressure Stack: Web (HTML/JS or the app's stack). Mode: challenge. Goal: Build a resilience test (chaos test) of the whole onboarding and use it to determine which defences are still missing. ## Context of the existing code You are working in an existing project (web application: a browser frontend, possibly with its own backend. If the project has a backend, the event store, broker and services belong on the backend, and the points about endpoints and tables refer to it. If it has no backend, use IndexedDB in the browser and `fake-indexeddb` in tests. Keep the project's framework and test runner; add new dependencies only with a justification.). Domain: a mortgage calculator that is gradually turning into an event-driven system (calculator, risk, approval, identity, account, documents, payout). Prerequisite: the project has asynchronous onboarding via a broker all the way to payout; the harness will show the missing defences. If the flow is missing, first create a minimal in-memory flow. Read first, change nothing. Find and briefly summarise: - Existing onboarding integration/e2e tests. - Fault-injection options: what takes time, timers, randomness and the network when the system is composed; inject faults and virtual time there as optional parameters (default behaviour unchanged). - Metrics: queues, DLQ, message age. - Which of these parts already exist in the project and which are missing. Before writing your own test infrastructure (virtual time, PRNG, fake broker), check whether the project or organisation already has it as a ready-made library, and use it. If it is missing, propose it in the plan. ## Expected behaviour With all faults at once, the defences complement each other (Outbox, retry and DLQ, Inbox and idempotency, correlation sequence, Saga) and none is enough on its own; with all of them the process completes consistently, just more slowly. Example: the “duplicate” profile (the document service writes the contract, the response is lost) finds two contracts for one application. ## Brief (Challenge) Write a reproducible chaos test of onboarding with post-run invariants and show which failures are not yet handled. Technical specifics (Web (HTML/JS or the app's stack)): Vitest with a deterministic PRNG (seed) and fake time: only the harness advances the clock and timers. The whole run is in a single process without a network (API, service HTTP handlers, in-memory broker); fake timers over real HTTP are not reproducible. Choose your own approach. The pitfalls in this topic are not listed: find them (concurrency, redelivery, ordering, failure midway) and for each write a test that would expose it. At the end, list which pitfalls you found and which you deliberately left open. ## What you must not change - The existing public API and the behaviour of the calculator endpoints that currently work. - The database schema of existing tables (new tables/migrations are fine; renaming and deleting are not). - Other people's tests, seed data and existing values. If a change is necessary, put it in a separate commit and state it. If a change to an event's shape or a new consumer necessarily changes an existing test, update it in a separate commit and explain why in the summary. - Public contracts (APIs, event schemas) without backward compatibility. ## Done when - The test is reproducible: two runs with the same seed give the same fingerprint; the seed is in the test name. - Invariants are explicit (in code and in the README). - Open risks are in the README, each with a test; the fixed gap has a test that failed before the fix. - Tests pass with the existing test runner; lint and build pass if the project has them. - In the summary, state: what you changed, what you assumed, what remains.
Related parts of the presentation
Training for your team
Would you like this training for your team?
We run the training on your code and your domain problem. We agree the scope, dates and format with you.
Discuss the training