Servers and erlangs
Requests reach a web server with four cores at random moments, about 320 a second. A core takes 10 ms per request on average. All four share one line. This is an M/M/4 queue: random arrivals, random service times, 4 servers.
Traffic intensity A = arrival rate × average service time = 320 × 0.01 s = 3.2 erlangs. One erlang is one server busy all the time, so on average 3.2 cores are busy: each is busy 3.2 ÷ 4 = 80% of the time.
A new request arrives while core 4 is free, so it starts at once. With c servers, an arrival waits only when all c are busy. The line also has to drain: A must stay below c, or it grows forever.
Erlang C
The next two requests find all four cores busy, so they wait in the shared line, first in, first out. Whichever core finishes first takes the request at the head of the line.
The Erlang C formula gives the chance that an arrival finds every server busy. For 4 servers and 3.2 erlangs it is 59.6%. One server alone at the same 80% load would make 80% of requests wait.
Core 1 finishes and takes the head of the line. Average wait = Erlang C ÷ (c × service rate − arrival rate) = 0.596 ÷ (400 − 320) per second = 7.46 ms. On average 2.39 requests wait (Little's law).
Pooling
Now give each core its own line and send each request to one at random. Each core becomes an M/M/1 queue at 80% load: 80% of requests wait, 40 ms on average. Here core 4 sits idle while two requests wait at core 1.
Back to one shared line: the idle core takes the next request at once. Same four cores, same load, but the average wait drops from 40 ms to 7.46 ms, 5.4 times shorter. No core idles while a request waits.
A help desk gets 64 calls an hour, 3 minutes each on average: 3.2 erlangs again. With 4 agents, 59.6% of callers are put on hold, for 2.2 minutes on average. A fifth agent cuts that to 29 seconds.
Keep every server 80% busy and grow the pool: the chance of waiting falls from 80% with 1 server to 59.6% with 4, 30.5% with 16 and 5.6% with 64. Big pools run busy and still answer fast.
The arithmetic
;; step 1/10 · many-servers · arrive ;; clients → shared queue Kendall's notation M / M / c M arrivals are memoryless: a Poisson process M service times are memoryless: exponential c servers sharing one line; here c = 4 arrival rate λ = 320 per second service rate μ = 1 / 10 ms = 100 per second per core
The figure shows a few requests at a time; the meter gives the long-run averages. The separate lines take requests at random. Sending each request to the shortest line helps, but it still loses to one shared line, because a request cannot switch to a core that frees up later.