collections and itertools essentials

What you'll learn

• Count things with Counter and read off the most common
• Group without checking for keys with defaultdict
• Use deque for queues and namedtuple for lightweight records
• Chain, batch, and combine iterables with itertools

Concept

The collections module holds specialized containers that replace patterns you have written by hand. Counter is a dictionary that counts: give it any iterable and it tallies each item, and most_common(n) returns the top entries sorted. Where you wrote counts[word] = counts.get(word, 0) + 1, Counter(words) does it in one call, and two counters add together with +.

defaultdict is a dictionary with a factory for missing keys. defaultdict(list) creates an empty list the first time you touch a new key, so grouping becomes groups[key].append(item) with no setdefault. defaultdict(int) starts missing keys at zero, which is another way to count. The factory is any zero-argument callable, so defaultdict(set) and even defaultdict(lambda: defaultdict(int)) for two-level counting are common.

deque, a double-ended queue, adds and removes from both ends in constant time. A list's pop(0) shifts every remaining item and gets slower as the list grows, so a deque is the right structure for a queue of tasks or a sliding window. Passing maxlen= makes it drop the oldest item automatically when a new one arrives, which turns it into a fixed-size window over a stream.

namedtuple creates a small tuple class with named fields. Point = namedtuple("Point", "x y") gives you p.x instead of p[0], immutability, and a readable repr, all in one line, and _asdict() converts an instance to a dictionary. It is the cheapest way to stop indexing records by magic number; the dataclasses lesson later covers its more capable cousin.

itertools supplies building blocks for loops over iterables. chain walks several iterables as one. batched splits a sequence into fixed-size chunks. product gives every combination of items from two or more iterables, permutations and combinations the arrangements and selections from one. groupby groups consecutive items with the same key, which is why you sort first. islice slices any iterable, including ones with no end, and count and cycle generate endless sequences to be sliced. Each replaces a nested loop or an index-juggling routine with a name that says what it does.

Code

Counter: from a word count to the top three, plus arithmetic between counters.

# counting.py
from collections import Counter

text = "the cat sat on the mat and the dog sat too"
counts = Counter(text.split())

print(counts.most_common(3))
print(counts["the"], counts["zebra"])

monday = Counter(coffee=3, tea=1)
tuesday = Counter(coffee=2, juice=1)
print(monday + tuesday)
print(Counter("mississippi").most_common(2))

[('the', 3), ('sat', 2), ('cat', 1)]
3 0
Counter({'coffee': 5, 'tea': 1, 'juice': 1})
[('i', 4), ('s', 4)]

defaultdict for grouping and two-level counting, and namedtuple for readable records.

# grouping.py
from collections import defaultdict, namedtuple

Sale = namedtuple("Sale", "region product amount")
sales = [Sale("north", "pen", 3), Sale("south", "ink", 8), Sale("north", "ink", 5), Sale("north", "pen", 2)]

by_region = defaultdict(list)
for sale in sales:
    by_region[sale.region].append(sale.product)
print(dict(by_region))

totals = defaultdict(lambda: defaultdict(int))
for sale in sales:
    totals[sale.region][sale.product] += sale.amount
print({region: dict(products) for region, products in totals.items()})

print(sales[0], sales[0].amount, sales[0]._asdict())

{'north': ['pen', 'ink', 'pen'], 'south': ['ink']}
{'north': {'pen': 5, 'ink': 5}, 'south': {'ink': 8}}
Sale(region='north', product='pen', amount=3) 3 {'region': 'north', 'product': 'pen', 'amount': 3}

deque as a queue and as a sliding window with maxlen.

# queues.py
from collections import deque

queue = deque(["task1", "task2"])
queue.append("task3")
print(queue.popleft(), list(queue))
queue.appendleft("urgent")
print(list(queue))

window = deque(maxlen=3)
for reading in [10, 12, 15, 11, 9]:
    window.append(reading)
    print(list(window), round(sum(window) / len(window), 1))

task1 ['task2', 'task3']
['urgent', 'task2', 'task3']
[10] 10.0
[10, 12] 11.0
[10, 12, 15] 12.3
[12, 15, 11] 12.7
[15, 11, 9] 11.7

itertools: chaining, batching, products, grouping consecutive runs, and slicing an endless counter.

# tools.py
from itertools import batched, chain, combinations, count, groupby, islice, product

print(list(chain([1, 2], (3,), "ab")))
print(list(batched(range(1, 8), 3)))
print(["".join(p) for p in product("ab", "12")])
print(list(combinations(["ada", "grace", "linus"], 2)))

words = sorted(["apple", "bee", "ant", "bat", "cow"], key=lambda w: w[0])
for letter, group in groupby(words, key=lambda w: w[0]):
    print(letter, list(group))

print(list(islice(count(10, 5), 4)))

[1, 2, 3, 'a', 'b']
[(1, 2, 3), (4, 5, 6), (7,)]
['a1', 'a2', 'b1', 'b2']
[('ada', 'grace'), ('ada', 'linus'), ('grace', 'linus')]
a ['apple', 'ant']
b ['bee', 'bat']
c ['cow']
[10, 15, 20, 25]

Try it yourself

Create log_stats.py with a list of twenty fake log lines like "ERROR db timeout", "INFO request ok", mixing three levels and a few components. Use Counter for the level totals and the two most common components, defaultdict(list) to collect messages per level, a deque(maxlen=5) to keep the last five lines, and batched to print the lines in groups of four.

Common mistakes

• Reading a missing key from a defaultdict only to check whether it exists. The read creates the key with an empty default; use in to test without inserting.
• groupby on unsorted data. It groups consecutive runs only; sort by the same key first or you get fragments.
• Reaching for deque everywhere. It is for the ends; indexing into the middle of a deque is slow, and a list is still the default sequence.

What's next

• Counter counts and ranks; defaultdict groups without key checks; deque gives fast queues and windows; namedtuple names tuple fields.
• itertools provides chain, batched, product, combinations, groupby (sort first), and islice over endless count.
• Each one replaces a hand-written loop with a name that states the intent.

Your standard-library toolkit is nearly complete. Next you will turn a script into a real command with flags and help text, using argparse.