Generators and lazy pipelines

What you'll learn

• Write a generator function with yield and understand how it pauses and resumes
• Use generator expressions for one-pass transformations
• Chain generators into a pipeline that processes data one item at a time
• Send a generator's output to sum, max, any, and friends without ever building a list

Concept

A generator is the easy way to write an iterator. Put yield in a function and it becomes a generator function: calling it runs nothing and returns a generator object, which is an iterator. Each next() runs the body until the next yield, hands out that value, and pauses with all local variables intact. When the body ends, the generator raises StopIteration for you. The Countdown class from the previous lesson, with its __iter__, __next__, and state tracking, becomes three lines.

That pause-and-resume is what makes generators lazy. Nothing runs until something asks for a value, and only enough runs to produce one. A generator can therefore describe an endless sequence, read a file of any size one record at a time, or do expensive work for the first three items and never touch the rest. Generator expressions, (x * 2 for x in items), are the one-line form: a comprehension with parentheses that yields instead of building a list.

The real power is chaining. Because a generator both consumes an iterator and is one, you can stack them: one reads lines, the next parses each line into a record, the next filters, the next transforms. Each item flows through the whole chain before the next is read, so a gigabyte file uses kilobytes of memory, and stopping early stops all the stages at once. This is how data processing is written in Python when the data does not fit in memory, and it reads as a list of small named steps.

Written out, each stage of that pipeline is one small generator, and the thing worth tracking is what each one hands to the next.

Because generators are iterators, the built-ins that accept any iterable consume them directly: sum, max, min, any, all, sorted, list, set, dict, "".join. Passing a generator expression straight into one of these is the idiomatic form you have already been using, sum(x.amount for x in items). The one thing to remember is that a generator is used up once; if you need the items twice, make a list, or call the generator function again to get a fresh one.

Code

The countdown as a generator, and the pause-and-resume made visible with prints inside the body.

# first_generator.py
def countdown(n):
    print("  (generator started)")
    while n > 0:
        yield n
        n -= 1
    print("  (generator finished)")


gen = countdown(3)
print(type(gen).__name__)
print("first:", next(gen))
print("second:", next(gen))
print("the rest:", list(gen))
print("again:", list(countdown(2)))

generator
  (generator started)
first: 3
second: 2
  (generator finished)
the rest: [1]
  (generator started)
  (generator finished)
again: [2, 1]

Endless and lazy. The generator never ends, islice takes what it needs, and the expensive step runs only for the items actually consumed.

# lazy_gen.py
import itertools


def squares():
    n = 1
    while True:
        yield n * n
        n += 1


def slow_double(items):
    for item in items:
        print(f"  doubling {item}")
        yield item * 2


print(list(itertools.islice(squares(), 6)))
doubled = slow_double([1, 2, 3, 4, 5])
print("first two only:", next(doubled), next(doubled))
print(any(x > 100 for x in squares()))

[1, 4, 9, 16, 25, 36]
  doubling 1
  doubling 2
first two only: 2 4
True

A pipeline over a CSV-like file: read, parse, filter, transform, and sum, without a single list. Each stage is a small named generator function.

# pipeline.py
from pathlib import Path

Path("orders.txt").write_text(
    "id,status,amount\n1,paid,12.50\n2,refunded,30.00\n3,paid,7.25\n4,paid,100.00\n5,pending,9.99\n",
    encoding="utf-8",
)


def read_lines(path):
    with open(path, encoding="utf-8") as f:
        next(f)                      # skip the header
        for line in f:
            yield line.rstrip("\n")


def parse(lines):
    for line in lines:
        order_id, status, amount = line.split(",")
        yield {"id": int(order_id), "status": status, "amount": float(amount)}


def only_paid(records):
    return (r for r in records if r["status"] == "paid")


def amounts(records):
    for r in records:
        yield r["amount"]


total = sum(amounts(only_paid(parse(read_lines("orders.txt")))))
print(f"paid total: {total:.2f}")

big = next(r for r in parse(read_lines("orders.txt")) if r["amount"] > 50)
print("first big order:", big["id"])

paid total: 119.75
first big order: 4

Generators feed the built-ins directly, and are used up once. The second sum over the same generator object is zero; a fresh call gives a fresh generator.

# consume.py
def scores():
    yield from [72, 95, 58, 88]


gen = scores()
print(sum(gen), sum(gen))
print(max(scores()), sorted(scores()), any(s > 90 for s in scores()))
print(", ".join(str(s) for s in scores()))
print({s: s >= 60 for s in scores()})

313 0
95 [58, 72, 88, 95] True
72, 95, 58, 88
{72: True, 95: True, 58: False, 88: True}

Try it yourself

Create logs.py that writes a file of 1,000 fake log lines, some containing ERROR. Build a pipeline of generators: read lines, keep only errors, extract the component name from each, and count them with Counter, without ever holding all lines in a list. Then write an endless fibonacci() generator and use itertools.takewhile to print every Fibonacci number under 1,000.

Common mistakes

• Expecting the body to run at call time. countdown(3) runs nothing; the body starts at the first next() or loop.
• Reusing a generator. After one full pass it is empty; call the function again or keep a list if you need the data twice.
• Returning a list from a function that could yield. If callers only loop once, yield keeps memory flat and lets them stop early; return a list only when they need it all at once.

What's next

• yield turns a function into a generator: each next() runs to the next yield and pauses with its state intact.
• Generator expressions are one-pass comprehensions; both feed sum, max, any, sorted, and join directly.
• Chain small generators into pipelines that process one item at a time and stop early for free.

You can process data lazily at any scale. Next you go back to functions to understand the rules that make closures and decorators possible: scope.