Lesson 46 · Building systems

Scaling out

When one server gets too busy, you can buy a bigger machine (scale up) or add more ordinary machines (scale out). Watch a site grow from one server to a row of stateless app servers behind a load balancer, see where the data goes instead (a database, a cache and object storage), how read replicas and shards spread the database work, where adding machines stops helping, and how autoscaling adds servers when the load rises.

12 min10 steps5 questionsBeginner

Fig 46 · Interactive

Loading figure…

or ← →

Check yourself

5 quick questions

Pick an answer to see why. Your score saves with your progress.

01What is the difference between scaling up and scaling out?

Scaling up (vertical) gives one machine more CPUs and memory. Scaling out (horizontal) adds more ordinary machines that share the work.

02A load balancer uses round robin over 3 app servers. Which server gets the 5th request?

Round robin hands requests out in turn: 1, 2, 3, 1, 2. Request r (counting from 0) goes to server (r mod 3) + 1, so request 5 goes to server 2.

03Why should app servers behind a load balancer be stateless?

The next request may land on a different server. If state lives in a shared cache or database, that server can still find it, and servers can be added or removed freely.

04A site's database is busy with reads, not writes. What helps most?

Read replicas are copies of the primary that can answer reads. The primary still takes every write. The copies can lag slightly behind.

05Autoscaling targets 60% CPU. 3 servers are at 80%. Using desired = ceil(current × metric ÷ target), how many servers should run?

ceil(3 × 80 ÷ 60) = ceil(4) = 4. The same total work spread over 4 servers puts each at 60%.

Key terms

Glossary

Scale up (vertical scaling): Making one machine bigger: more CPUs, memory or disk. Simple, but limited by the largest machine, and still one point of failure.
Scale out (horizontal scaling): Adding more machines that run the same program and share the work.
Load balancer: A single public address that passes each incoming request to one of several servers, and skips servers that fail their health check.
Round robin: Handing requests to servers in turn: 1, 2, 3, then 1 again.
Stateless: A server that keeps nothing between requests, so any copy of it can answer any request.
State: Data that must outlast one request, such as accounts, orders, sessions and uploaded files.
Read replica: A read-only copy of a database that receives every change from the primary and answers reads.
Sharding: Splitting one database's data across several databases (shards) by a key, so each takes only part of the writes.

Fine print

Notes and sources

The CPU percentages are an example load chosen to make the arithmetic easy to follow, not a measurement of a real site.
Round robin is the default in common load balancers such as nginx; others pick the server with the fewest open connections, or hash the visitor address. A load balancer can itself become a limit, so large sites run several.
Some setups pin each visitor to one server (sticky sessions) instead of sharing the session. The Twelve-Factor App calls that a violation of stateless design: when that server goes away, so does the session.
Replicas here copy changes asynchronously, which is why reads can lag. Synchronous replication avoids the lag but makes every write wait for the copies.
In practice each shard usually has its own replicas too, and moving data between shards (rebalancing) is real work. Ranged sharding by name can also leave one shard busier than another.
Kubernetes skips a change when the ratio is within a tolerance (10% by default) of 1, and waits before removing servers, so the count does not flap. Here 80 / 60 = 1.33, well past it.

Quiz Unlocks after the last step.