Why many cores
For decades each new chip ran a faster clock. Around 2004 that stopped near 3 to 4 GHz (billion ticks per second). A chip's power grows with its clock speed and with its voltage squared, so faster chips got too hot to cool.
Chips still gained more transistors each year, so makers spent them on more cores. A core is a complete processor: it fetches and runs its own instructions. Four cores can run four programs, or four parts of one program, at the same time.
Caches that agree
Each core has private L1 and L2 caches: small, fast memories that keep copies of data the core used recently. L1 answers in about a nanosecond. Only their own core uses them, so cores never queue for them.
In front sits one bigger L3 cache that all four cores share, and a bus: shared wires that every core's caches listen to. Data that is not in a core's private caches comes over the bus from L3, or from main memory.
Core 0 reads a variable, x. Its private caches miss, so the request goes over the bus and L3 sends x = 5. Core 0's cache keeps a copy marked E, exclusive: no other cache holds x.
Now core 2 reads x too and gets its own copy. Two caches hold x, so both copies are marked S, shared: several caches may read it, and every copy says 5.
Core 0 writes x = 6. First it sends an invalidate message on the bus. Core 2's cache drops its copy (I, invalid), so it can never read the old 5. Core 0's copy becomes M, modified: the only up-to-date one.
Core 2 reads x again. Its copy is invalid, so that is a miss. Core 0's cache sees the request on the bus and sends 6, also writing it back to L3. Both copies are S again. That is cache coherence.
How much faster?
Most programs have a serial part that only one core can do. Here 1 of 10 time units is serial and 9 can be split. With four cores those 9 take 2.25: 1 + 2.25 = 3.25 units, about 3.1 times faster.
More cores shrink only the parallel part. With 16 cores: 1 + 9/16 ≈ 1.56 units. With endless cores, the serial 1 unit still remains. So a program that is 90% parallel can never run more than 10 times faster.
Hyper-threading (simultaneous multithreading) lets one core hold two threads: two streams of instructions. When one waits for memory, the other keeps the core busy. It helps (Intel measured up to about 30% more), but it is not a second core.
Numbers and code
;; step 1/11 · power-wall ;; about: Clock speed year chip clock power 1993 Pentium 66 MHz 10.1 W 1997 Pentium Pro 200 MHz 29.1 W 2001 Pentium 4 (Willamette) 2000 MHz 75.3 W 2004 Pentium 4 (Prescott) 3600 MHz 103 W 2007 Core 2 (Kentsfield) 2667 MHz 95 W 2010 Core i5 (Clarkdale) 3333 MHz 87 W 2012 Core i5 (Ivy Bridge) 3400 MHz 77 W power ∝ capacitive load × voltage² × frequency The clock stopped climbing; the power could not go higher and still be cooled.
Dynamic power is proportional to capacitive load × voltage² × switching frequency. Intel's x86 clock rates rose for about 25 years to 3.6 GHz (Pentium 4 Prescott, 2004, 103 W) and then flattened while power fell back below 100 W (Patterson and Hennessy, Computer Organization and Design, section 1.7, The Power Wall; Sutter, The Free Lunch Is Over, 2005). Amdahl's law: speed-up = 1 / ((1 − p) + p / n) for a parallel fraction p on n cores (Amdahl, AFIPS 1967). With p = 0.9: 4 cores give 3.08×, 16 cores 6.4×, and no number of cores can exceed 1 / 0.1 = 10×.