Compilers and interpreters up close
The CPU only runs machine code, so every program you write has to be translated. Follow one line of Python, return len(alist), as a lexer cuts it into tokens and a parser builds a syntax tree. Then see the three roads from there: a compiler such as clang writes machine code ahead of time, CPython writes bytecode that its virtual machine runs, and a JIT compiles the hot code while the program runs. Each road trades start-up time, speed and portability differently.
5 quick questions
Pick an answer to see why. Your score saves with your progress.
01What does the lexer (tokenizer) produce from return len(alist)?
Lexing only groups characters into tokens, the words of the language. Building the tree is the parser's job.
02Where does a missing closing bracket, as in return len(alist, get caught?
The tokenizer reaches the end of the file with ( still open, and the parser, which checks the tokens against the grammar, reports SyntaxError: '(' was never closed. Nothing runs.
03In CPython's dis output for myfunc, which instruction pushes the local variable alist onto the stack?
LOAD_FAST pushes a local variable. LOAD_GLOBAL finds len, CALL calls it and RETURN_VALUE hands back the result.
04Why can't the arm64 machine code clang wrote run on an x86-64 Intel PC?
Machine code is made of one chip family's instructions. The same C compiled for x86-64 gives different instructions (lea and ret).
05What does a JIT compiler such as V8 or PyPy do?
A JIT starts at once like an interpreter, notices which code runs again and again, and compiles just that code at run time.
Glossary
- Compiler
- A program that translates source code into another form, such as machine code (gcc, clang) or bytecode (CPython's compiler).
- Interpreter
- A program that runs code directly, one instruction or statement at a time, instead of producing a machine-code file first.
- Token
- One word of a program, such as a name, a number or a bracket. The lexer (tokenizer) cuts the source text into tokens.
- Syntax tree
- A tree the parser builds from the tokens. It records what the code means, such as 'return the result of calling len with alist'.
- Machine code
- Numbered instructions a CPU runs directly. Each chip family (arm64, x86-64) has its own.
- Bytecode
- Simple instructions for a virtual machine instead of a real chip. CPython's bytecode is shown by the dis module and cached in .pyc files.
- Virtual machine (VM)
- A program that runs bytecode in a loop: fetch an instruction, do it, move to the next.
- JIT compiler
- A just-in-time compiler: it compiles the often-run (hot) parts of a program to machine code while the program runs.
Notes and sources
- The figure tokenizes line 2 only. Python's tokenize module labels keywords such as return as NAME; the parser tells keywords from other names. The whole function also gives NEWLINE, INDENT and DEDENT tokens, because indentation is part of Python's grammar.
- The tree is simplified from what ast.parse returns for line 2: Return(value=Call(func=Name(id='len'), args=[Name(id='alist')])). CPython compiles the whole function, def line included. With return len(alist and no closing bracket, the tokenizer reaches the end of the file with ( still open and the parser reports SyntaxError: '(' was never closed: the lexer and the parser work together.
- The bytecode is CPython 3.13's, as in the dis documentation. Bytecode changes between versions: CPython 3.14 prints LOAD_FAST_BORROW instead of LOAD_FAST. (len + NULL) means LOAD_GLOBAL also pushes a NULL that CALL uses.
- The machine code is Apple clang 21 -O2 output for int add(int a, int b) { return a + b; } on arm64. For x86-64 the same compiler writes lea eax, [rdi + rsi] and ret. The C and Python examples are different functions.
- CPython is mostly a bytecode interpreter, but 3.13 added an experimental JIT that is off by default (PEP 744). Real engines mix the roads: V8 compiles JavaScript to bytecode first, and gcc can also stop at assembly text (-S).
- Python 3.13 docs: dis, disassembler for Python bytecode
- Python docs: tokenize, tokenizer for Python source
- Python docs: ast, abstract syntax trees
- Python glossary: bytecode
- Python tutorial 6.1.3: compiled Python files
- GCC manual: options controlling the kind of output (compilation stages)
- Arm A64 instruction set: ADD (shifted register)
- V8 blog: Firing up the Ignition interpreter
- V8 blog: Sparkplug, a non-optimizing JavaScript compiler
- PyPy documentation: introduction
- PEP 744: JIT compilation
- R. Nystrom, Crafting Interpreters: A Map of the Territory
- Stanford CS143: Compilers
Quiz Unlocks after the last step.