Generators and context managers¶
In 1 minute
- A generator makes values one at a time, when asked, instead of building a whole list first. It uses almost no memory.
- A function with
yieldis a generator: it hands out a value, pauses, and carries on from there when the next value is wanted. - Chain generators into a pipeline (read → parse → filter), and only one order is in memory at any moment, even for millions.
- A generator can be used once. Loop over it a second time and it's empty.
- A context manager runs setup when a
withblock starts and clean-up when it ends, even after an error.open()is one;@contextlib.contextmanagermakes your own.
Previously in Bazaar: decorators added timing and retries to any function in one line.
1. Why learn this?¶
Bazaar's order log has grown to 200,000 lines (and it will be millions). The finance team wants the number and total of paid orders.
The Module 4 way, "read every line into a list, then add up", builds 200,000 objects in memory at once. On a big Diwali day, that list can be bigger than the server's RAM, and the program dies.
But the sum only ever needs one order at a time. Generators let Python work like that. And context managers make sure files (and later, database connections) are always closed, whatever happens.
2. The concept¶
yield: hand out a value, then pause¶
"""A generator pauses at each yield and carries on from there when asked for the next value."""
def order_numbers():
print(" (generator starts)")
yield 1001 # hand out a value, then PAUSE here
print(" (generator resumes)")
yield 1002
print(" (generator resumes again)")
yield 1003
print(" (generator ends)")
numbers = order_numbers() # nothing runs yet: you get a generator object
print(type(numbers).__name__)
print(next(numbers)) # run until the first yield
print(next(numbers)) # carry on until the next yield
for n in numbers: # a for loop calls next() for you, until the end
print(n)
generator
(generator starts)
1001
(generator resumes)
1002
(generator resumes again)
1003
(generator ends)
- A function with
yieldin it is a generator function. Calling it runs nothing: it gives you a generator object. next(numbers)runs the function until the nextyield, hands out that value, and pauses there, remembering every local name.- The next
next()carries on from the pause. Aforloop simply callsnext()again and again, until the function ends.
Why it matters: memory¶
"""A list holds every value at once. A generator makes them one at a time, so it hardly uses memory."""
import sys
as_list = [n * 2 for n in range(1_000_000)] # square brackets: build all 1,000,000 values now
as_gen = (n * 2 for n in range(1_000_000)) # round brackets: a generator, values made on request
print("list: ", sys.getsizeof(as_list), "bytes")
print("generator:", sys.getsizeof(as_gen), "bytes")
print("same total?", sum(as_list) == sum(as_gen))
The same million numbers: 8.4 MB as a list, 200 bytes as a generator. The generator doesn't store the numbers; it only knows how to make the next one. Round brackets instead of square ones turn a comprehension (Module 4) into a generator expression.
A pipeline of generators¶
"""Reading 200,000 orders with a pipeline of generators: only ONE line is in memory at a time."""
def read_lines(path):
with open(path, encoding="utf-8") as f:
for line in f: # a file is already a generator of lines
yield line.rstrip("\n")
def parse(lines):
for line in lines:
number, customer, product, price, status = line.split(",")
yield {"customer": customer, "product": product, "price_paise": int(price), "status": status}
def paid_only(orders):
for order in orders:
if order["status"] == "PAID":
yield order
paid = paid_only(parse(read_lines("data/orders.log"))) # nothing has been read yet
count = 0
total = 0
for order in paid: # now the lines flow through, one by one
count += 1
total += order["price_paise"]
print(f"paid orders: {count:,}")
print(f"paid total: ₹{total // 100:,}")
Read the line paid = paid_only(parse(read_lines(...))): three generators, plugged into each other like pipes. Nothing is read yet.
When the for loop asks for an order:
paid_onlyasksparsefor an order,parseasksread_linesfor a line,read_linesreads one line from the file and yields it, then everything pauses again.
One line travels through the whole pipeline before the next is even read. The program uses the same small amount of memory for 200 orders or 200 million.
Used once, then empty¶
A generator remembers where it paused. When it reaches the end, it stays at the end. A second loop over the same generator gets nothing. That's this lesson's exercise (part 4). If you need the values twice, either keep them in a list (if they fit), or do both jobs in one loop.
Context managers: with always cleans up¶
You've written with open(...) as f: since Module 4. with works with any context manager: an object that runs setup at the start of the block
and clean-up at the end, whatever happens inside. You can make your own:
"""A context manager runs setup code at the start of `with` and clean-up code at the end, even after an error."""
import contextlib
@contextlib.contextmanager
def till_open(cashier):
print(f"{cashier} opens the till") # setup: runs when `with` starts
try:
yield # the body of the `with` block runs here
finally:
print(f"{cashier} closes the till") # clean-up: runs at the end, ALWAYS
with till_open("Neha"):
print(" sold 2 Masala Box")
try:
with till_open("Rohan"):
print(" selling Wireless Earbuds...")
raise ValueError("card declined")
except ValueError as error:
print("error:", error)
Neha opens the till
sold 2 Masala Box
Neha closes the till
Rohan opens the till
selling Wireless Earbuds...
Rohan closes the till
error: card declined
@contextlib.contextmanagerturns a generator with oneyieldinto a context manager: code beforeyieldis the setup, code after is the clean-up.try/finallyaround theyieldmakes the clean-up run even when the block raises: Rohan's card was declined, and the till still closed.
That's exactly why with open(...) closes the file even if your code crashes halfway. Module 16 uses the same idea for database transactions.
An analogy¶
A list is a printed telephone directory: every number is printed, bound and carried, even if you only need one.
A generator is calling enquiry (directory assistance): you ask, they tell you one number, and wait for your next question. They remember where you got to, but they don't hand you the whole book. And once you've gone through to the end, the call is over.
A context manager is the shop shutter rule: whoever opens the shutter in the morning must pull it down at night, even if the day goes badly.
Questions you may have¶
Is a generator slower than a list? Barely, and often faster overall, because nothing big has to be built first. Choose by memory, not speed.
When should I still use a list? When the data is small and you need it more than once, or need len(), sorting, or data[5].
Isn't a file already a generator of lines? Yes: for line in f reads one line at a time. read_lines just wraps it, so the pipeline reads nicely.
3. See it in action¶
Pick a list or a generator, then press Play. Watch the MEMORY box.
4. Code example¶
The lab's exercise: the average paid order value. It crashes:
"""Average PAID order value from the log. BUG: it crashes with ZeroDivisionError.
Your task: find why `count` is 0, and fix it (hint: a generator can be used up only once)."""
def paid_amounts(path):
with open(path, encoding="utf-8") as f:
for line in f:
number, customer, product, price, status = line.rstrip("\n").split(",")
if status == "PAID":
yield int(price)
amounts = paid_amounts("data/orders.log")
total = sum(amounts)
count = sum(1 for _ in amounts)
print(f"average paid order: ₹{total // count // 100:,}")
Traceback (most recent call last):
File "/lab/exercises/average_order.py", line 16, in <module>
print(f"average paid order: ₹{total // count // 100:,}")
~~~~~~^^~~~~~~
ZeroDivisionError: integer division or modulo by zero
sum(amounts) walked the generator to the end. The second walk, sum(1 for _ in amounts), found nothing left: count is 0.
The fix does both jobs in one pass:
total = 0
count = 0
for amount in paid_amounts("data/orders.log"): # one pass: both numbers at once
total += amount
count += 1
print(f"average paid order: ₹{total // count // 100:,}")
Still one order in memory at a time, and the file is read only once.
5. Under the hood¶
Asking past the end¶
Traceback (most recent call last):
File "/lab/mistakes/stop_iteration.py", line 8, in <module>
print(next(numbers)) # nothing left
~~~~^^^^^^^^^
StopIteration
StopIteration is how a generator says "no more". A for loop catches it for you and simply stops; next() by hand doesn't.
Generators have no length¶
A generator doesn't know how many values it will make until it has made them. Count while looping (count += 1), or use a list if it's small.
Going deeper: islice, yield from, and enter/exit
"""Going deeper: islice, yield from, and a context manager written as a class."""
import itertools
def read_lines(path):
with open(path, encoding="utf-8") as f:
yield from f # "yield every value of f, one by one"
first_three = list(itertools.islice(read_lines("data/orders.log"), 3)) # take only 3 lines, then stop reading
print([line.split(",")[1] for line in first_three])
class Till:
"""The same idea as @contextlib.contextmanager, written with __enter__ and __exit__."""
def __init__(self, cashier):
self.cashier = cashier
def __enter__(self): # runs when `with` starts; its return value goes after `as`
print(f"{self.cashier} opens the till")
return self
def __exit__(self, exc_type, exc, tb): # runs at the end, always; gets the error if there was one
print(f"{self.cashier} closes the till (error: {exc_type.__name__ if exc_type else None})")
return False # False: don't hide the error
with Till("Neha") as till:
print(" ", till.cashier, "is selling")
itertools.islice(gen, 3)takes the first 3 values and stops: the rest of the 200,000-line file is never read.itertoolshas many generator tools (chain,groupby,batchedin 3.12+).yield from fyields every value of another iterable: a short way to writefor line in f: yield line.- A context manager class has
__enter__(setup; its return value is whatasgets) and__exit__(clean-up; it receives the error, if any). ReturningTruefrom__exit__would swallow the error. Almost never do that.
Going deeper: generators that receive values, and where else you'll meet them
value = yield xlets the caller send a value in withgen.send(...). This is how Python'sasyncio(Lesson 8) began.- Django's
QuerySet.iterator()streams database rows one by one instead of loading them all (Module 10). - A web server can stream a big CSV export as a generator, so the download starts before the whole file exists.
6. Lab¶
Lab folder: labs/05-advanced-python/05-generators-and-context-managers/
Generate the 200,000-line order log, watch a generator pause at each yield, compare a list's memory with a generator's, sum the paid orders through a pipeline,
see a till close even after an error, and fix the average that crashes because a generator was used twice.
When you finish, run bash test_lab.sh (from your computer). You should see ALL CHECKS PASSED.
7. Common mistakes¶
- Looping over a generator twice. The second loop is empty (here, a
ZeroDivisionError). Fix: one pass, or a list if the data is small. [ … ]when you meant( … ). A million-item list instead of a generator. Fix: round brackets insidesum(),any(), afor.len()or[0]on a generator.TypeError. Fix: count while looping, orlist(...)small data first.- Opening files without
with. An error halfway leaves them open. Fix: alwayswith open(...). - Clean-up without
finallyin a@contextmanager. An error skips it. Fix:try: yield/finally:clean-up.
8. Quiz¶
9. What's next¶
Next: giving each project its own set of installed packages, and turning Bazaar into an installable package: Virtual environments and packaging.
In one line: a generator yields values one at a time and pauses in between, so pipelines over millions of lines use almost no memory;
it can be used only once; a context manager (with) always runs its clean-up, even after an error.
Cheat sheet¶
| Task | Code |
|---|---|
| A generator function | def read_lines(path): … yield line |
| A generator expression | (o["price_paise"] for o in orders) |
| Next value by hand | next(gen) (StopIteration at the end) |
| A pipeline | paid_only(parse(read_lines(path))) |
| Only the first N | itertools.islice(gen, 3) |
| Your own context manager | @contextlib.contextmanager + try: yield / finally: … |
| Use it | with till_open("Neha"): … |
Next in Bazaar: Bazaar's code now needs mypy, and soon Django and more. Install them all into your computer's one Python, and two projects will fight over versions. Next lesson: virtual environments, and packaging Bazaar properly.