Up or out
Every visitor's request lands on one app server, the machine running the site's program. As visitors grow, its processor (CPU) gets busier: here 90%. Near 100%, new requests start waiting in line, and pages slow down.
Scaling up, or vertical scaling, means moving to a bigger machine with more CPUs and memory. The code does not change, so it is the easy first move. But even the biggest machine has a limit, and if it fails, the whole site stops.
Scaling out, or horizontal scaling, means adding more ordinary machines, each running the same program. 3 servers share the work, so each is only 30% busy. Need more? Add another one. If one breaks, the others keep going.
Stateless servers
Visitors now reach one address: the load balancer. It hands each request to the next server in turn: 1, 2, 3, 1. This is called round robin. It also stops sending requests to any server that fails its health check.
Your next request may land on a different server. So app servers are stateless: they keep nothing between requests. Server 1 saves your login session in a shared cache, and server 2 reads it from there.
Scaling the data
State is any data that must outlast one request. It lives in shared stores every server can reach: a database for records like orders, a cache for fast, short-lived data like sessions, and object storage for files like photos.
Most sites read far more than they write. The primary database takes every write and copies it to read replicas, and reads can go to any copy. The copy can lag a moment behind, so a fresh write may not show yet.
When writes outgrow one primary, split the data. Each shard holds part of it, chosen by a key. Here users A to M live on shard 1 and N to Z on shard 2, so each write goes to just one shard.
Limits and autoscaling
More servers only speed up work that can run side by side. Amdahl's law: if 10% of the work must wait its turn at one shared part, like the database, no number of servers makes it more than 10 times faster.
Autoscaling adds and removes servers by itself. With a 60% CPU target, 3 servers at 80% become 3 × 80 ÷ 60 = 4, each back to 60%. When traffic drops, it removes servers again, so you pay for less.
What happens
;; step 1/10 · one-server visitors → app server 1 CPU busy 90% (example load) near 100% requests queue and pages slow down the site needs more computing power
The CPU percentages are an example load chosen to make the arithmetic easy to follow, not a measurement of a real site. Some setups pin each visitor to one server (sticky sessions) instead of sharing the session. The Twelve-Factor App calls that a violation of stateless design: when that server goes away, so does the session.