Part 16 of 16 · Day 2 · end-to-end, chaos testing
Bugs that only appear in combination
Problem and experience
Everything at once, in front of the client
The client sees only the result. Operations see queues, the DLQ and delays. Both should see the truth.
Failure
- Lost publication
- Service errors
- Duplicates
- Out of order
- Lost ack
From practiceWhich of these failures have you already experienced in production?
TakeawayEach mechanism works on its own. Problems show up only in combination.
What it means
The whole process involves the client (operation status), operations (metrics, DLQ) and all services at once.
Why we use it
We want to know whether what we have learnt holds together.
When it makes sense
Before going live and after larger integration changes.
Where it breaks
- Testing only the happy path.
- Not verifying both the client's view and the operations view.
Imagine
With 25 % publication loss and 50 % duplicates, the process passes only with Outbox, retry, Inbox and ordering together.
A system is not resilient until we have tried to beat it.
Principle, application and exercise
End-to-end and chaos test
We run the whole process with injected faults. We judge success by invariants, not by nothing having crashed.
Defence
- Outbox
- Retry + DLQ
- Inbox
- corrSeq
- Saga
Invariants after completion
- Exactly one account
- Exactly one payout
- View = source of truth
Trade-offA chaos test costs time and needs a fixed seed to be repeatable. Turning on every defence “just in case” without understanding what each one solves is not a solution.
TakeawayLet's turn on defences one at a time and watch which cause each one removes.
What it means
A chaos test runs the system with injected faults and checks invariants after completion.
Why we use it
The individual defences complement one another. None is enough on its own.
When it makes sense
Before going live, after an integration change, and regularly.
Where it breaks
- A test that fails once in a hundred runs: use a fixed seed.
- Measuring only success and ignoring unwanted side effects.
Imagine
The process completes, but with a duplicate account. The test fails on the “exactly one account” invariant.
No defence is enough on its own. We ask which cause each of them removes.
Training for your team
Would you like this training for your team?
We run the training on your code and your domain problem. We agree the scope, dates and format with you.
Discuss the training →