Design for failure
Users buy through a shop service, which calls a payments service. Machines crash, networks drop and services slow down, every day. Reliability means users still get a working shop, so you design for failure from the start.
Run two copies of the shop on different machines. When copy A crashes, requests go to copy B and users see no gap. A spare only helps if both copies cannot fail together.
Payments goes slow. Without a limit, the shop would wait forever while users pile up. A timeout gives up after a set time, here 2 seconds, and the shop answers at once: please try again soon.
Many failures last a moment, so try again, but wait first. Each wait may be twice as long: up to 0.1, then 0.2, then 0.4 s (exponential backoff). A random pick below that cap (jitter) stops clients retrying all at once. Try 3 works.
Now payments is down, and retries would only add load. After 5 failures in a row the breaker opens: calls fail at once without reaching payments. Users get a quick "try later", and payments gets room to recover.
After a cool-down, say 30 seconds, the breaker goes half-open and lets one test call through. Payments answers, so the breaker closes and calls flow again. Had it failed, the breaker would open for another cool-down.
Measure what users feel
To see inside a running system, services send out three signals. Logs are timestamped records of events. Metrics are numbers over time, like requests and errors per second. A trace follows one request through every service and shows where time went.
Measure what users feel. An SLI (service level indicator) is one such number: here, the share of requests that succeed. An SLO (objective) is its target: 99.9% over 30 days. This month it is 99.94%, so the SLO is met.
A 99.9% SLO lets 0.1% fail: 1,000 of this month's 1,000,000 requests, or about 43 minutes of full outage. The payments outage used 600, leaving 40%. Budget left: ship features. Budget gone: fix reliability first.
When it breaks
Page a person only for symptoms users feel, not for every busy CPU. Here the rule is: errors are using up the budget fast, 2% of it in one hour. The lamp lights and the on-call engineer is paged.
In an incident one person coordinates, and the team restores service first. Afterwards a blameless postmortem records what happened, the impact, and why the system allowed it, then lists fixes with owners. It asks what failed, not who.
What happens
;; step 1/11 · things-fail users → shop → payments → shop → users every part can fail: a machine crashes the network drops packets a service it depends on gets slow reliability: users still get a working shop
The 2-second timeout, the 5-failure threshold, the 30-second cool-down, the traffic and the size of the outage are example values; real ones are tuned for each service. The arithmetic on them is exact. Requests reach a healthy copy through a load balancer, which checks each copy and stops sending it traffic when it fails. The figure leaves the load balancer out.