Inside a data center
A data center is a big room full of servers stacked in tall cabinets called racks. On top of each rack sits a switch for its servers, called a leaf. Here: four racks, servers A to D, and leaves 1 to 4.
One click on a website can make a web server ask a database, a cache and a search server for parts of the answer. That server-to-server traffic is called east-west. It is far bigger than north-south traffic to and from the internet.
Leaf and spine
Behind the racks stand spine switches. Every leaf has one cable to every spine. Leaves never connect to other leaves, and spines never connect to other spines. This pattern is called leaf-spine, a kind of Clos network.
Server A talks to server D: up to leaf 1, across a spine, down to leaf 4. Server B to server C has the same shape. Any two servers in different racks are exactly three switches apart, so delay is short and predictable.
Many equal paths
Leaf 1 can reach leaf 4 through spine 1 or through spine 2. Both paths are the same length. Equal-cost multipath (ECMP) lets the leaf use both at once, instead of keeping one idle as a spare.
For each packet, leaf 1 mixes header fields (addresses and ports) into a number called a hash, and the hash picks a spine: spine 1. Every packet of one flow gives the same hash, so the flow keeps one path and stays in order.
A flow is one conversation, such as one database query. Different flows have different addresses or ports, so they hash differently and land on different spines. These four flows share both spines, two each.
Growing and routing
Need more room between racks? Add a spine and cable it to every leaf. Now there are three equal paths, and each leaf has half as much capacity again. Need more servers? Add a rack whose leaf cables to every spine.
How does leaf 1 learn those paths? Many data centers run BGP on every cable (RFC 7938). Each leaf has its own AS number and announces its rack's addresses. Leaf 1 hears leaf 4's 10.0.4.0/24 from all three spines, equally good.
On top, many data centers add an overlay. Leaf 1 wraps a frame in an outer packet addressed to leaf 4 (VXLAN, RFC 7348). The spines route only the outer packet, and leaf 4 unwraps it. Machines in different racks share one virtual network.
Inside the fabric
;; step 1/10 · racks · idle server A 10.0.1.10 rack 1, under leaf 1 (10.0.1.0/24) server B 10.0.2.10 rack 2, under leaf 2 (10.0.2.0/24) server C 10.0.3.10 rack 3, under leaf 3 (10.0.3.0/24) server D 10.0.4.10 rack 4, under leaf 4 (10.0.4.0/24) a leaf is also called a top-of-rack (ToR) switch
The hash shown (XOR of the IP address and port bytes, modulo the number of spines) is an example. Real switches use their own formulas; what matters is that one flow always gets the same path. The 100 Gbit/s uplink speed is an example; real fabrics use many speeds.