Many servers: M/M/c and Erlang C
A web server with four cores, a call centre with four agents: many servers share one line. Measure traffic in erlangs, use the Erlang C formula to find the chance that a request has to wait and how long it waits, then see why one shared line beats a separate line for each server.
5 quick questions
Pick an answer to see why. Your score saves with your progress.
01Requests arrive at 600 per second and each takes 5 ms of a server's time on average. How much traffic is that?
A = arrival rate × mean service time = 600 × 0.005 s = 3 erlangs: on average three servers are busy all the time.
02What does the Erlang C formula give you?
Erlang C is P(wait) for an M/M/c queue with unlimited waiting room. The chance of being turned away when there is no waiting room is Erlang B.
03With 4 servers, 320 arrivals a second and a 10 ms mean service time, Erlang C is 0.596. What is the mean wait in the line?
Wq = C ÷ (cμ − λ) = 0.596 ÷ (4 × 100 − 320) per second = 7.46 ms. 12.5 ms is the mean wait of only those requests that do wait.
04Four servers are each 80% busy. Why does one shared line give much shorter waits than four separate lines?
With separate lines a request can wait behind a busy server while another server is idle. Pooling removes that: here the mean wait drops from 40.0 ms to 7.46 ms, about 5.4 times shorter.
05Keep every server 80% busy but grow the pool from 4 to 64 servers. What happens to the chance of waiting?
Bigger pools absorb random bursts better: P(wait) is 59.6% with 4 servers, 30.5% with 16 and 5.6% with 64, all at 80% utilisation.
Glossary
- M/M/c
- Kendall's notation for a queue with random (Poisson) arrivals, random (exponential) service times and c servers sharing one first-in, first-out line.
- Server
- Whatever does the work: a CPU core, a worker thread, a link, or an agent in a call centre.
- Erlang
- The unit of traffic intensity: one erlang keeps one server busy all the time. A = arrival rate × mean service time (ITU-T E.600).
- Utilisation (ρ)
- The share of time each server is busy: ρ = A ÷ c. It must stay below 1, or the line grows without limit.
- Erlang C
- The probability that an arrival finds all c servers busy and has to wait, for a system with unlimited waiting room.
- Erlang B
- The probability that an arrival finds all servers busy when there is no waiting room, so it is lost (blocked).
- Pooling
- Letting several servers share one queue, so no server idles while work waits. At the same load it cuts waiting sharply.
- Little's law
- Average number waiting = arrival rate × average wait (L = λW). It holds for any stable queue.
Notes and sources
- The 320 requests a second, 10 ms per request and the 64 calls an hour of 3 minutes are illustrative, chosen to give 3.2 erlangs; every other number follows from the M/M/c formulas.
- M/M/c assumes random (Poisson) arrivals and exponential service times. Real traffic is often burstier, which makes waits longer, but the pooling effect still holds.
- The separate lines take requests at random. Sending each request to the shortest line helps, but it still loses to one shared line, because a request cannot switch to a core that frees up later.
- The figure shows a few requests at a time; the meter gives the long-run averages.
- Erlang B is the related formula for systems with no waiting room, where a call that finds every line busy is lost.
- Harchol-Balter, Performance Modeling and Design of Computer Systems (Cambridge UP, 2013), ch. 14: M/M/k and resource pooling
- Kleinrock, Queueing Systems, Volume 1: Theory (Wiley, 1975), §3.5: M/M/m
- MIT OpenCourseWare 6.262: Discrete Stochastic Processes (Gallager)
- ITU-T E.600: Terms and definitions of traffic engineering
- Little, A Proof for the Queuing Formula: L = λW (Operations Research 9(3), 1961)
Quiz Unlocks after the last step.