Iterators and the iteration protocol

What you'll learn

• Explain what a for loop actually calls: iter() and next()
• Tell an iterable from an iterator, and why a list can be looped twice but an iterator cannot
• Write a class that supports for by implementing __iter__ and __next__
• Recognize lazy iteration and why it matters for large or endless data

Concept

A for loop looks like magic: hand it a list, a string, a dictionary, a file, or an object you wrote, and it walks the items. Underneath is one small protocol. The loop calls iter(thing) to get an iterator, then calls next(iterator) repeatedly, receiving one item each time, until next raises StopIteration. That is the whole mechanism, and everything that supports for supports it by implementing those two functions.

The two words matter. An iterable is anything iter() accepts: lists, strings, dictionaries, files, ranges. An iterator is what iter() returns: an object with a __next__ method that remembers its position. A list is iterable but not an iterator; calling iter(list) gives a fresh iterator each time, which is why you can loop over the same list twice. An iterator is consumed as it goes: once it has raised StopIteration, it is finished, and looping over it again yields nothing. Files, zip, map, enumerate, and generator expressions all return iterators, and that is the source of the "why is my second loop empty" bug.

To make your own class loopable, give it __iter__. The simplest version returns an iterator over some internal collection, return iter(self._items), and that is enough for most classes. To control the iteration yourself, write a separate iterator class with __next__ that tracks state and raises StopIteration when done, or, far more often, use a generator, which the next lesson introduces and which writes the iterator for you.

Iterators, and lazy sequences like range, produce one item when asked and no more. That is why range(10**9) is instant, why a file can be read line by line without loading it, and why zip and map cost nothing until you loop. Laziness also allows endless iterators, like itertools.count(), which would be impossible to build as a list. When you see a function that returns an iterator instead of a list, that is a deliberate choice: it lets the caller stop early and keeps memory flat.

Code

The protocol by hand. iter gives an iterator, next pulls items, and the exhausted iterator raises StopIteration, which for catches for you.

# protocol.py
colors = ["red", "green", "blue"]
it = iter(colors)

print(type(it).__name__)
print(next(it), next(it), next(it))
print(next(it, "no more"))
next(it)

list_iterator
red green blue
no more
Traceback (most recent call last):
  File "protocol.py", line 7, in <module>
    next(it)
    ~~~~^^^^
StopIteration

Iterable versus iterator: a list loops twice, a zip iterator is consumed once and empty the second time. Check the kind with iter(x) is x.

# exhausted.py
pairs_list = list(zip("ab", [1, 2]))
pairs_iter = zip("ab", [1, 2])

print([p for p in pairs_list], [p for p in pairs_list])
print([p for p in pairs_iter], [p for p in pairs_iter])

for thing in (pairs_list, pairs_iter, "text", open(__file__)):
    it = iter(thing)
    print(f"{type(thing).__name__:<20} iterator itself? {it is thing}")

[('a', 1), ('b', 2)] [('a', 1), ('b', 2)]
[('a', 1), ('b', 2)] []
list                 iterator itself? False
zip                  iterator itself? True
str                  iterator itself? False
TextIOWrapper        iterator itself? True

Your own iterable, two ways. Playlist delegates to its list; Countdown is a full iterator with __next__ and its own state.

# custom.py
class Playlist:
    def __init__(self, songs):
        self._songs = list(songs)

    def __iter__(self):
        return iter(self._songs)


class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1


for song in Playlist(["intro", "verse"]):
    print(song)
print(list(Countdown(3)), list(Countdown(3)))
cd = Countdown(2)
print(list(cd), list(cd))

intro
verse
[3, 2, 1] [3, 2, 1]
[2, 1] []

Laziness in action. The range and the file are never materialized, an endless counter is safe to create, and zip stops as soon as the shortest input does.

# lazy.py
import itertools
import sys
from pathlib import Path

big = range(10**9)
print(len(big), sys.getsizeof(big), "bytes")

Path("lines.txt").write_text("\n".join(f"line {i}" for i in range(100_000)), encoding="utf-8")
with open("lines.txt", encoding="utf-8") as f:
    for number, line in enumerate(f, start=1):
        if number == 3:
            print("stopped early at:", line.strip())
            break

endless = itertools.count(1)
print(list(itertools.islice(endless, 5)))
print(list(zip(endless, "abc")))

1000000000 48 bytes
stopped early at: line 2
[1, 2, 3, 4, 5]
[(6, 'a'), (7, 'b'), (8, 'c')]

Try it yourself

Create iterate.py with a Fibonacci iterator class whose __next__ yields the next Fibonacci number forever, and use itertools.islice to print the first ten. Then write a Deck class that stores 52 card strings and supports for, len(), and in. Finally, create one map object, loop over it twice, and explain in a comment why the second loop printed nothing.

Common mistakes

• Looping over an iterator twice. zip, map, files, and generators are consumed; wrap in list() if you need the items again.
• Forgetting StopIteration. A __next__ that never raises it produces an infinite loop; make sure some state ends it.
• Materializing everything. list(f) on a huge file or list(range(n)) for a large n loads it all; loop lazily and stop early when you can.

What's next

• for calls iter() once and next() until StopIteration; that protocol is all any loopable object needs.
• Iterables produce fresh iterators; iterators are consumed once, which explains the empty second loop.
• Laziness keeps memory flat and allows endless sequences; implement __iter__ (and __next__) to join in.

You know what iteration is. Next you will write iterators the easy way: generators, with yield, and chain them into lazy pipelines.