Lesson 63 · Network design

Data center: leaf-spine

Inside a data center, most traffic flows between servers, not to the internet. Watch racks of servers join a leaf-spine fabric where every leaf switch connects to every spine, so any two racks are three switches apart. See a hash spread flows over equal paths (ECMP), a new spine add capacity, BGP learn the paths and VXLAN carry a virtual network across it all.

9 min10 steps5 questionsBeginner

Fig 63 · Interactive

Loading figure…

or ← →

Check yourself

5 quick questions

Pick an answer to see why. Your score saves with your progress.

01A web server asks a database server in the same data center for some data. What kind of traffic is that?

Traffic between servers inside the data center is called east-west. North-south is traffic to and from the outside, such as users on the internet.

02In a leaf-spine fabric, how many switches does a packet cross from a server in rack 1 to a server in rack 4?

Every leaf connects to every spine, so the path is always up to the source leaf, across one spine and down to the destination leaf: three switches for any two racks.

03Leaf 1 has two equal paths to leaf 4. Why does it hash header fields instead of sending packets on the paths in turn?

The same flow always gives the same hash, so it always takes the same path and its packets cannot overtake each other. Different flows get different hashes and spread over the paths.

04A fabric has 4 leaves and 2 spines. You add a third spine. How many leaf-to-spine cables are there now?

Every leaf connects to every spine: 4 leaves × 3 spines = 12 cables. The new spine adds one cable per leaf and a third equal path between every pair of leaves.

05What does VXLAN do in a data center fabric?

The source leaf wraps the frame in an outer IP and UDP packet (port 4789) addressed to the destination leaf. The spines route only the outer packet, and the far leaf unwraps it.

Key terms

Glossary

East-west traffic: Traffic between servers inside a data center. North-south traffic goes in and out, to and from the internet.
Leaf: The switch at the top of a rack that its servers plug into, also called a top-of-rack (ToR) switch. In a leaf-spine fabric every leaf connects to every spine.
Spine: A switch in the middle layer of a fabric. Spines connect only to leaves, never to each other.
Clos network: A network of switches in stages where every switch in one stage connects to every switch in the next, described by Charles Clos in 1953. Leaf-spine is a folded Clos network.
ECMP: Equal-cost multipath: when several paths are equally good, use them all, picking one per flow with a hash of header fields.
Flow: One conversation between two programs, such as one database query; its packets share the same addresses and ports.
AS number: A number that identifies a routing domain to BGP. In RFC 7938 fabrics all spines share one and every leaf has its own, from the private range.
VXLAN: Virtual Extensible LAN (RFC 7348): carries Ethernet frames inside UDP packets between leaves, so a virtual network can stretch across many racks.

Fine print

Notes and sources

Leaf-spine is a folded Clos network, named after Charles Clos, who described multi-stage switching networks for telephone exchanges in 1953.
The hash shown (XOR of the IP address and port bytes, modulo the number of spines) is an example. Real switches use their own formulas; what matters is that one flow always gets the same path.
RFC 7938 describes eBGP on every leaf-spine link: all spines share one AS number (65100 here), every leaf has its own (65001-65004), from the private range 64512-65534 (RFC 6996).
VXLAN (RFC 7348) carries Ethernet frames inside UDP port 4789. Its 24-bit network identifier allows about 16 million virtual networks; with IPv4 the wrapping adds 50 bytes.
The 100 Gbit/s uplink speed is an example; real fabrics use many speeds.
Addresses come from the private ranges 10.0.0.0/8 and 172.16.0.0/12 (RFC 1918), MACs from the RFC 7042 documentation block, source ports from the dynamic range 49152-65535 (RFC 6335).
Packets cross a data center in microseconds; the figure slows everything down so you can watch.

Quiz Unlocks after the last step.