Lesson 21 · Idiomatic Python

Iterators and generators

Every for loop in Python is driven by an iterator: an object that hands out one value per next() call and raises StopIteration when it has no more. A generator is the easiest way to write one: a function with yield in it. Step through countdown.py and watch the generator pause after each yield, keep its local n and its place, carry on exactly where it stopped on the next call, and finish with StopIteration, which the for loop catches quietly.

14 min9 steps5 questionsBeginner

Fig 21 · Interactive

Loading figure…

or ← →

Check yourself

5 quick questions

Pick an answer to see why. Your score saves with your progress.

01gen = countdown(3) has just run. What has the body of countdown() printed or done so far?

Calling a generator function only builds the generator object (state GEN_CREATED). The body starts running at the first next() call.

02After two calls of next(gen) on countdown(3), what is the value of n inside the generator?

The first next() yields 3 and pauses. The second resumes after the yield: n -= 1 makes n 2, and yield hands out 2. The generator keeps n = 2 while it is paused.

03How does a for loop know when to stop looping over an iterator?

A for loop calls iter() once and then next() on each pass. When next() raises StopIteration, the loop ends quietly, with no error shown.

04gen = countdown(2); print(list(gen)); print(list(gen)) — what is printed?

A generator runs only once. The first list() uses it up; the second gets StopIteration straight away and builds an empty list.

05Why can sum(x * x for x in range(10**6)) use far less memory than sum([x * x for x in range(10**6)])?

Both give 333332833333500000. The list comprehension builds a list of a million squares before sum() starts; the generator expression is lazy and hands sum() one square at a time.

Key terms

Glossary

Iterable: Anything you can loop over with for, such as a list, a string, a dict or a file. iter() turns it into an iterator.
Iterator: An object that hands out values one at a time: next() returns the next one, and raises StopIteration when there are none left.
next(): A built-in function that asks an iterator for its next value.
StopIteration: The exception an iterator raises when it has no more values. A for loop catches it and simply ends.
Generator function: A def whose body contains yield. Calling it does not run the body; it returns a generator.
Generator: The iterator a generator function returns. Each next() runs its body up to the next yield, then pauses, keeping its local variables.
yield: Hands a value out of a generator to whoever called next(), and pauses the generator on that line until it is asked again.
Lazy evaluation: Making each value only when it is asked for, instead of building all of them up front.

Fine print

Notes and sources

An iterable is anything iter() accepts, such as a list, a string or a dict; an iterator is the object iter() returns. An iterator has a __next__() method (called by next()) and an __iter__() method that returns the iterator itself, which is why iter(gen) is gen.
The "next line" box is a picture of what Python keeps for a paused generator: its local variables and where in the body to carry on. While a generator is suspended, gen.gi_frame.f_lineno reports the yield line itself (line 3); it carries on with the line after it.
A generator runs only once. When it is finished, every further next() raises StopIteration at once, and a for loop over it does nothing. Call countdown(3) again to get a fresh one.
The state names come from inspect.getgeneratorstate(): GEN_CREATED, GEN_RUNNING, GEN_SUSPENDED and GEN_CLOSED.
Inside a generator, return (or reaching the end of the body) raises StopIteration for you. Raising StopIteration yourself inside a generator is turned into a RuntimeError since Python 3.7 (PEP 479).

Quiz Unlocks after the last step.