14.1 Iterables and Iterators
Two related but distinct ideas get conflated constantly:
| Term | What it is | Requirement |
|---|---|---|
| Iterable | Anything you can loop over with for — something you can get an iterator from |
implements __iter__() (13.1) |
| Iterator | The object that actually produces items one at a time, remembering where it is | implements both __iter__() (returning itself) and __next__() |
list, str, tuple, dict, and set are all iterables — but none of them is its own
iterator. Calling iter() on one hands back a separate iterator object that does the
actual work:
nums = [1, 2, 3]
it = iter(nums) # a fresh iterator, separate from the list itself
print(type(nums)) # <class 'list'>
print(type(it)) # <class 'list_iterator'>
print(next(it)) # 1
print(next(it)) # 2
print(next(it)) # 3
try:
next(it) # nothing left - the iterator raises StopIteration to signal "done"
except StopIteration:
print("StopIteration raised - iterator exhausted")
<class 'list'> <class 'list_iterator'> 1 2 3 StopIteration raised - iterator exhausted
Common mistake — assuming a list and its iterator behave the same way. An
iterable like a list can be looped over as many times as you like, because each for
gets a brand-new iterator. An iterator, once exhausted, stays exhausted forever — even
if you still hold a reference to it:
nums = [1, 2, 3]
for x in nums:
pass
for x in nums: # works again - the list handed out a NEW iterator this time
print(x, end=" ")
print() # 1 2 3
it = iter(nums)
list(it) # consumes the iterator fully
print(list(it)) # [] - this exact iterator object is now permanently exhausted
1 2 3 []
In practice — Django's QuerySet is lazy in exactly this way. Writing
User.objects.filter(active=True) doesn't touch the database at all — it builds up a
description of the query, the same way a generator function builds a paused computation
without running it. The actual SELECT only fires once the queryset is iterated (a
for loop, list(...), etc.), which is why chaining several .filter()/.exclude()
calls together costs nothing extra until the moment the results are actually consumed.
14.2 The Iterator Protocol
A for loop is syntax sugar — it never inspects the object's type directly. It calls
iter(obj) once to get an iterator, then calls next() on that iterator repeatedly
until a StopIteration exception tells it to stop (7.4 — StopIteration is a plain
built-in exception, caught exactly like any other). iter(obj) and next(it) are
themselves just syntax for the dunder calls type(obj).__iter__(obj) and
type(it).__next__(it) (13.1's a + b → __add__ mechanism, applied here). Writing
the loop out by hand makes the whole mechanism visible:
nums = [10, 20, 30]
# exactly what "for value in nums:" does under the hood
it = iter(nums)
while True:
try:
value = next(it)
except StopIteration:
break
print(value) # 10 20 30 - same result as a plain for loop
10 20 30
14.3 Custom Iterators
Any class can plug into for loops the same way built-ins do — implement __iter__
(returning self, since the object is its own iterator here) and __next__
(returning the next value, or raising StopIteration when there's nothing left):
class Countdown:
def __init__(self, start):
self.current = start
def __iter__(self):
return self # this object is its own iterator
def __next__(self):
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
for n in Countdown(3):
print(n) # 3 2 1
3 2 1
Common mistake — treating a custom iterator like a reusable iterable. Because
__iter__ returns self instead of a fresh object, a Countdown instance is its own
iterator — so once it's exhausted, it's exhausted for good, exactly like the bare
list_iterator in 14.1's gotcha, and unlike the Countdown class itself (each new
Countdown(3) call still starts fresh):
c = Countdown(3)
print(list(c)) # [3, 2, 1] -> first pass, consumes it fully
print(list(c)) # [] -> self.current is already 0, same object, nothing left
[3, 2, 1] []
A more reusable pattern — __iter__ returning a fresh iterator. Countdown above
is its own iterator (__iter__ returns self), which is why it can only be looped
over once. Most real-world iterables — range, list, dict — instead give __iter__
a body that produces a brand-new iterator (or generator) on every call, keeping no
shared state on the object itself. That's what lets the same object be iterated over
repeatedly, exactly like the list in 14.1:
class CountdownReusable:
def __init__(self, start):
self.start = start
def __iter__(self):
current = self.start
while current > 0: # a fresh generator each call - no shared state to exhaust
yield current
current -= 1
reusable = CountdownReusable(3)
print(list(reusable)) # [3, 2, 1] -> first pass
print(list(reusable)) # [3, 2, 1] -> works again, unlike the single-use Countdown
[3, 2, 1] [3, 2, 1]
14.4 Generator Functions
Writing a custom iterator class (14.3) by hand is a lot of boilerplate just to remember
"where was I." A generator function — any function containing a yield statement —
gets the compiler to write that class for you. Calling it doesn't run the function body
at all; it immediately returns a generator object (an iterator that already has
__iter__/__next__ built in). The body only starts running on the first next()
call, and it runs up to the next yield, then pauses there — with every local
variable and its exact position frozen — and hands back the yielded value. The next
next() call resumes exactly where it left off. When the function finally returns (or
just runs off the end), Python raises StopIteration automatically, the same signal a
custom __next__ raises by hand:
def count_up_to(n):
print("starting")
i = 1
while i <= n:
yield i
i += 1
print("done")
gen = count_up_to(3)
print(type(gen)) # <class 'generator'> -> body hasn't run at all yet
print(next(gen)) # starting \n 1 -> runs up to the first yield, then pauses
print(next(gen)) # 2 -> resumes right after that yield
print(next(gen)) # 3
try:
next(gen) # nothing left to yield -> runs the rest of the body, then StopIteration
except StopIteration:
print("StopIteration") # done \n StopIteration
<class 'generator'> starting 1 2 3 done StopIteration
Rule of thumb. In practice, almost all custom iteration is written as a generator
function like the one above; the hand-written class form from 14.3
(__iter__/__next__ implemented directly) matters mainly for understanding the
mechanism a generator function desugars to, not as the everyday tool.
List vs. generator — the whole point of yield. A function that returns a list
builds the entire result in memory before handing it back. A generator function
builds nothing — it produces one value at a time, on demand, and the generator object
itself stays essentially the same tiny size no matter how many values it will
eventually yield:
import sys
def squares_list(n): # eager - builds the whole list right now
result = []
for i in range(n):
result.append(i ** 2)
return result
def squares_gen(n): # lazy - computes each square only when asked
for i in range(n):
yield i ** 2
big_list = squares_list(1_000_000)
big_gen = squares_gen(1_000_000)
print(sys.getsizeof(big_list)) # 8448728 -> one full million-item array, already built
print(sys.getsizeof(big_gen)) # 200 -> just the paused-function bookkeeping, that's all
print(sys.getsizeof(big_gen) < sys.getsizeof(big_list)) # True -> holds no matter how large n gets
8448728 200 True
In practice — processing a multi-gigabyte log or CSV export. A data-engineering
script that scans a huge log file for a pattern, or streams a large CSV export row by
row into a database, is exactly this trade-off in action: reading the whole file into a
list first could exhaust memory long before processing even starts, while a generator
(or the file object's own line-by-line iteration, 16.5) processes one row at a time and
uses roughly the same small amount of memory whether the file is 10MB or 10GB.
Caveat. The exact byte counts above (8448728, 200) are specific to this CPython
version and build — they'll differ on other versions or interpreters (PyPy, etc.), so
don't treat them as fixed constants. What's guaranteed, not just observed here, is the
shape of the relationship: the generator's size stays constant regardless of n,
while the list's grows with it.
Fibonacci via yield is the classic showcase, because it makes the laziness
concrete: a plain function would have to decide up front how many terms to compute and
build a whole list of them, but a generator can produce terms one at a time forever,
using a fixed, tiny amount of state (a, b) no matter how far it's asked to go:
def fibonacci(n): # the first n Fibonacci numbers - a bounded generator
a, b = 0, 1
for _ in range(n):
yield a
a, b = b, a + b
print(list(fibonacci(10))) # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
Drop the bound entirely (while True instead of for _ in range(n)) and it becomes an
infinite generator — something a list-returning function could never be, since
building an infinite list would never finish. itertools.islice (9.6) pulls off just
as many terms as needed and stops, without the generator ever knowing it was "infinite":
def fibonacci_infinite(): # never returns - would hang forever if you tried list(...)
a, b = 0, 1
while True:
yield a
a, b = b, a + b
import itertools
first_eight = list(itertools.islice(fibonacci_infinite(), 8))
print(first_eight) # [0, 1, 1, 2, 3, 5, 8, 13] -> only 8 terms were ever computed
[0, 1, 1, 2, 3, 5, 8, 13]
In practice — a paginated API client. Consuming a REST API that returns results a
page at a time is a very common real use of exactly this pattern: a generator function
fetches and yields one page's worth of items, then transparently fetches the next
page only when the caller asks for more — the caller just loops over it like any other
iterable, never needing to know or care how many pages exist or manage the pagination
tokens itself.
14.5 Generator Expressions
A generator expression is to a generator function what a list comprehension (5.1.3)
is to a for loop with .append() — the same for ... in ... shape, but with ()
instead of [], and lazy instead of eager. Swapping the brackets is the entire
difference — one builds a list immediately, the other builds a paused generator that
computes nothing until it's consumed:
squares_list_comp = [x ** 2 for x in range(5)] # eager - the list exists right now
squares_gen_expr = (x ** 2 for x in range(5)) # lazy - nothing computed yet
print(squares_list_comp) # [0, 1, 4, 9, 16]
print(squares_gen_expr) # <generator object <genexpr> at 0x...>
print(list(squares_gen_expr)) # [0, 1, 4, 9, 16] -> NOW it runs, once
print(list(squares_gen_expr)) # [] -> already exhausted, just like any other generator
[0, 1, 4, 9, 16] <generator object <genexpr> at 0x7f2de798e670> [0, 1, 4, 9, 16] []
A generator expression that's the only argument to a function doesn't even need its
own parentheses — sum(x ** 2 for x in range(...)) is idiomatic, and never builds an
intermediate list at all, no matter how large the range:
total = sum(x ** 2 for x in range(1_000_000)) # no intermediate list of a million squares
print(total) # 333332833333500000
333332833333500000
In practice — a reporting/aggregation query over records. sum(order.total for
order in orders if order.paid)-style aggregation directly over a stream of records
(rows from a database cursor, 18.3, or lines from a log file) is a common shape in
analytics code — the generator expression never materializes the intermediate list of
matching orders, it just feeds sum() one value at a time.
14.6 Advanced Generator Behaviour
yield isn't only a way to send values out — it's also an expression, which
means a value can be sent in at the same point the generator paused, via
generator.send(value). send() resumes the generator exactly like next() does, but
whatever was sent becomes the result of the yield expression itself inside the
function. generator.close() stops it early, raising GeneratorExit inside and then
making it permanently exhausted:
def running_total():
total = 0
while True:
value = yield total # pauses here; "value" becomes whatever send() provides
if value is not None:
total += value
acc = running_total()
print(next(acc)) # 0 -> must "prime" it once to reach the first yield
print(acc.send(10)) # 10 -> value=10 resumes the loop, total becomes 10, then pauses again
print(acc.send(5)) # 15 -> value=5, total becomes 15
acc.close() # stops the generator for good
try:
next(acc)
except StopIteration:
print("StopIteration - generator closed")
0 10 15 StopIteration - generator closed
Common mistake — calling .send() before priming the generator. A freshly-created
generator hasn't started running yet — it hasn't reached its first yield, so there's
no paused yield expression for a sent value to become. The first call has to be
next(gen) (or the equivalent gen.send(None)) to prime it up to that first yield;
only after that can .send(value) deliver a real value in:
def running_total():
total = 0
while True:
value = yield total
if value is not None:
total += value
fresh = running_total()
try:
fresh.send(10) # no yield reached yet - nothing to send a value INTO
except TypeError as e:
print("TypeError:", e) # can't send non-None value to a just-started generator
TypeError: can't send non-None value to a just-started generator
yield from delegates to another iterable, yielding all of its values as if they
were yielded directly — the tool for composing generators without writing a manual
inner loop:
def inner():
yield 1
yield 2
def outer():
yield "start"
yield from inner() # equivalent to "for v in inner(): yield v"
yield "end"
print(list(outer())) # ['start', 1, 2, 'end']
['start', 1, 2, 'end']
Common mistake — letting StopIteration escape from inside a generator's body.
A StopIteration raised inside a generator — as opposed to the one Python raises
automatically when the generator finishes — is not allowed to propagate out silently.
Python converts it into a RuntimeError instead, because an uncaught StopIteration
looks exactly like "this generator is done," and letting it leak would silently and
incorrectly terminate whatever for loop or yield from chain is consuming the
generator. The usual way this bites: calling next() on an inner iterator without
catching the StopIteration it raises once exhausted:
def broken_generator():
inner = iter([1, 2])
while True:
yield next(inner) # once `inner` is exhausted, this raises StopIteration...
gen = broken_generator()
print(next(gen)) # 1
print(next(gen)) # 2
try:
next(gen) # ...which Python converts into a RuntimeError, not StopIteration
except RuntimeError as e:
print("RuntimeError:", e) # generator raised StopIteration
1 2 RuntimeError: generator raised StopIteration
Going deeper. This same send/close/yield from machinery — priming, resuming with a
value, propagating GeneratorExit on close() — is exactly what powers
@contextlib.contextmanager (16.6) under the hood: it wraps a single-yield generator
function so it can double as a context manager, translating with block entry/exit into
send()/close()/throw() calls on that generator. In everyday code you'll reach for
that decorator constantly and only rarely write raw .send()/.close() calls yourself.
14.7 Laziness, Memory and Performance
Pulling the chapter's comparisons together — when to reach for a list (or a list
comprehension) versus a generator (function or expression):
list (eager) |
Generator (lazy) | |
|---|---|---|
| Memory | O(n) — every item exists at once (14.4) | O(1) — one item's worth of state, regardless of n |
| Speed to first item | Waits for the whole collection to build | Immediate — only the first item is computed |
| Re-iterable? | Yes, as many times as you like (14.1) | No — single-use, exhausted after one pass (14.1, 14.3) |
len(), indexing (x[i]), slicing |
Supported | Not supported (below) |
| Works with infinite sequences? | No — would never finish building | Yes (14.4's fibonacci_infinite) |
A generator's laziness is also its biggest limitation: since items aren't stored anywhere, there's no way to ask "how many?" or "give me item 3" without consuming the whole thing to find out — so neither operation is supported at all:
gen = (x for x in range(5))
try:
len(gen)
except TypeError as e:
print("TypeError:", e) # object of type 'generator' has no len()
try:
gen[0]
except TypeError as e:
print("TypeError:", e) # 'generator' object is not subscriptable
TypeError: object of type 'generator' has no len() TypeError: 'generator' object is not subscriptable