The levels
A computer keeps data in several kinds of memory, stacked here like a pyramid. The top levels are tiny but very fast. Each level down holds much more, costs less per byte, and takes much longer to answer.
Registers sit inside the processor core, the part that runs instructions. An instruction reads a register in under a nanosecond (ns, a billionth of a second). But there are only a few dozen of them: about a kilobyte in all.
Next come three caches, L1, L2 and L3, made of SRAM (static RAM, about six transistors per bit). L1 answers in about 1 ns but holds only around 64 KB. L3 is bigger, tens of megabytes, and slower: 10 to 20 ns.
Main memory (RAM) is DRAM: one transistor and one tiny capacitor per bit, so it packs gigabytes cheaply. But a trip to RAM takes about 100 ns, a hundred times longer than L1.
SSDs and hard disks keep data with the power off. An SSD (flash) answers in tens of microseconds. A hard disk must spin its platter and move its arm, so it takes milliseconds: roughly 100,000 times slower than RAM.
A request falls down
The core needs a value from memory. It asks L1 first. L1 has a copy, so this is a hit: the value comes back in about 1 ns and lands in a register.
At first, the value is in none of the caches. L1 misses, so the request falls to L2, which misses, then L3, which misses too. RAM has it: a hit at last, about 100 ns after the core asked.
On its way back up, the value left a copy in L3, L2 and L1. Ask for the same value again and L1 hits: about 1 ns instead of 100.
Why caches work
Why ask again? Programs reuse data. In this loop, total is read and written on every pass. That is temporal locality: data used recently is likely to be used again soon. So it stays in L1, and nearly every use hits.
The loop also reads nums in order, item after item. That is spatial locality: data next to recent data is likely next. So a miss brings a whole 64-byte cache line, eight of the list's 8-byte slots, and the next several reads hit.
Thanks to locality, most requests hit the small, fast levels. Say 95% hit L1 and 5% go all the way to RAM. Every request pays L1's 1 ns, misses also pay 100 ns: average 1 + 0.05 × 100 = 6 ns.
Numbers and code
;; step 1/11 · pyramid ;; about: Registers, L1 cache, L2 cache, L3 cache, RAM (DRAM), SSD (flash), Hard disk level time size registers <1 ns ~1 KB L1 cache ~1 ns ~64 KB L2 cache 3–10 ns ~1 MB L3 cache 10–20 ns tens of MB RAM (DRAM) 50–100 ns gigabytes SSD (flash) ~50 µs ~1 TB hard disk 5–10 ms 4–16 TB 1 ns = a billionth of a second 1 µs = 1,000 ns · 1 ms = 1,000,000 ns (typical numbers; the drawing is not to scale)
Typical sizes and access times after Hennessy and Patterson, Computer Architecture: A Quantitative Approach (6th ed., figure 2.1): registers about 1000 bytes and 300 ps, L1 64 KB at about 1 ns, L2 3-10 ns, L3 10-20 ns, main memory 50-100 ns, flash 25-50 microseconds, disk 5-10 ms. Real chips vary; AMD's Ryzen 9 7950X has 1 MB of L2 per core and 64 MB of L3. Average memory access time = hit time + miss rate × miss penalty (Hennessy and Patterson, appendix B). With a 1 ns hit, a 5% miss rate and a 100 ns penalty: 1 + 0.05 × 100 = 6 ns.