Lesson 47 · Building systems

Reliability and observability

Every part of a real system fails sometimes: machines crash, networks drop and services slow down. Follow a web shop that keeps working anyway, with spare copies, timeouts, retries that back off with a random wait, and a circuit breaker. Then see how teams measure what users feel with logs, metrics, traces, SLIs, SLOs and an error budget, alert a person only when users are hurt, and learn from incidents without blame.

12 min11 steps5 questionsBeginner

Fig 47 · Interactive

Loading figure…

or ← →

Check yourself

5 quick questions

Pick an answer to see why. Your score saves with your progress.

01One checkout took 3 seconds. Which signal shows where that one request spent its time, service by service?

A trace follows one request through every service and times each step (a span). Logs record single events, and metrics are numbers summed up over many requests, so neither shows one request's path.

02Why do retries wait a random time below a cap that doubles each try (exponential backoff with jitter)?

Waiting longer each time gives a struggling service room to recover, and the random part spreads clients out so their retries do not arrive together.

03A service has an SLO of 99.9% success over 30 days and gets 1,000,000 requests in that time. How many may fail?

The error budget is 100% minus the SLO: 0.1%. And 0.1% of 1,000,000 is 1,000 requests.

04Which alert should wake someone up at night?

Page people for symptoms users feel. A busy CPU or a single retry may not hurt anyone; errors burning the budget fast do.

05Payments is fixed. The team now writes a blameless postmortem. What is it for?

A postmortem writes down the impact, the timeline, the root causes and follow-up actions. Blameless means it looks at the system and the process, not at who to punish, so people report problems honestly.

Key terms

Glossary

Redundancy: Running spare copies of a part, so that when one fails another carries on.
Timeout: A limit on how long to wait for an answer. When it runs out, the caller gives up and handles the failure.
Exponential backoff and jitter: Retrying after waits whose limit doubles each time, with a random wait below that limit so clients do not retry together.
Circuit breaker: Code that stops calling a failing service for a while (open), then lets a test call through (half-open) before closing again.
Logs, metrics, traces: The three signals of observability: timestamped event records, numbers measured over time, and the path of one request through every service.
SLI and SLO: A service level indicator measures what users get, such as the share of successful requests; a service level objective is its target, such as 99.9% over 30 days.
Error budget: How much failure the SLO allows (100% minus the SLO). Teams ship changes while it lasts and fix reliability when it runs out.
Blameless postmortem: A write-up after an incident that records what happened, its impact, why the system allowed it, and fixes, without blaming people.

Fine print

Notes and sources

The 2-second timeout, the 5-failure threshold, the 30-second cool-down, the traffic and the size of the outage are example values; real ones are tuned for each service. The arithmetic on them is exact.
The card uses "full jitter" from the AWS article on backoff: wait a random time between 0 and the cap. A real version would also limit the longest wait, skip the wait after the last try, and report the error if every try fails.
Requests reach a healthy copy through a load balancer, which checks each copy and stops sending it traffic when it fails. The figure leaves the load balancer out.
43.2 minutes is 0.1% of 30 days (43,200 minutes). Counting requests and counting minutes are two ways to measure the same budget.
The fast-burn page comes from the Google SRE Workbook: a burn rate of 14.4 spends 2% of a 30-day budget in one hour.

Quiz Unlocks after the last step.