Debugging

What you'll learn

• Stop a running program at a line with breakpoint() and inspect its state in pdb
• Use the debugger commands that matter: step, next, continue, print, and where
• Debug a failing test with pytest's --pdb and in VS Code with breakpoints
• Debug systematically: reproduce, narrow, inspect, fix, and add a test

Concept

Debugging by print works until it does not: the value you need is three calls deep, the bug happens on the 400th iteration, or the state you want to see is an object with twelve fields. A debugger pauses the program at a line you choose and lets you look at everything, run expressions, and move forward one line at a time. Python's built-in one is pdb, and breakpoint() anywhere in your code opens it at that line. Delete the call when you are done; it is a tool, not a fixture.

A handful of commands cover almost every session. p expr prints a value, pp pretty-prints it, n runs the current line and stops at the next, s steps into a function call, c continues to the next breakpoint or the end, l lists the code around you, w shows the call stack (where am I, and who called me), u and d move up and down that stack to inspect a caller's variables, and q quits. Any other input is evaluated as Python in the current scope, so you can call functions and test hypotheses on the live state.

Tests are the best place to debug. pytest --pdb drops into the debugger at the point a test fails, with the failing frame active, so p shows the actual values the assertion saw. pytest -x --pdb stops at the first failure. In VS Code, click the gutter to set a breakpoint, press F5 with the Python debugger configured, and you get the same power with a graphical variables pane and a call stack view; the "Python Debugger" extension's "Debug Test" links do this per test. The environment variable PYTHONBREAKPOINT=0 disables every breakpoint() call, which is a safety net for one that reaches production.

The debugger is one step in a method. Reproduce the bug reliably, ideally as a failing test. Narrow it: bisect the input, cut the code path in half, find the smallest case that still fails. Inspect at the boundary where good state becomes bad, which is where the debugger earns its place. Fix the cause, not the symptom. Then keep the test so the bug cannot return. Most debugging time is spent in the first two steps, and the debugger makes the third step fast rather than replacing the others.

Code

A pdb session driven from a script so the transcript is real. The commands a person would type are fed through standard input, so they are not echoed after the (Pdb) prompts; read the stdin block alongside the output: print a variable, step to the next line, print again, and continue.

# session.py
def running_total(values):
    total = 0
    for value in values:
        breakpoint()
        total += value
    return total


print("result:", running_total([3, 4]))

p total, value
n
p total
c
c

> session.py(4)running_total()
-> breakpoint()
(Pdb) (0, 3)
(Pdb) > session.py(5)running_total()
-> total += value
(Pdb) 0
(Pdb) > session.py(4)running_total()
-> breakpoint()
(Pdb) result: 7

The bug you would use a debugger for: the answer is wrong on one particular input and a print at the top level cannot see why. w shows how execution got there and u inspects the caller's variables.

# where.py
def normalize(scores):
    top = max(scores)
    return [s / top for s in scores]


def report(groups):
    results = {}
    for name, scores in groups.items():
        if name == "empty":
            breakpoint()
        results[name] = normalize(scores) if scores else []
    return results


print(report({"a": [2, 4], "empty": [], "b": [1]}))

p name, scores
w
c

> where.py(10)report()
-> breakpoint()
(Pdb) ('empty', [])
(Pdb)   where.py(15)<module>()
-> print(report({"a": [2, 4], "empty": [], "b": [1]}))
> where.py(10)report()
-> breakpoint()
(Pdb) {'a': [0.5, 1.0], 'empty': [], 'b': [1.0]}

Debugging a failing test. pytest --pdb opens the debugger in the failing frame; here the session prints the values the assertion compared and quits.

# tests/test_totals.py
def total_with_tax(prices, rate):
    return sum(prices) * 1 + rate


def test_total_with_tax():
    assert total_with_tax([10, 20], 0.1) == 33.0

# debug_test.sh
printf 'p sum([10, 20]) * 1 + 0.1\np (10 + 20) * (1 + 0.1)\nq\n' | pytest -q --pdb tests/test_totals.py 2>&1 | grep -E "^\(Pdb\)|^30|^33|assert 30"

E       assert 30.1 == 33.0
(Pdb) 30.1
(Pdb) 33.0
(Pdb)
FAILED tests/test_totals.py::test_total_with_tax - assert 30.1 == 33.0

Disabling breakpoints without editing code, for the one that reaches production, and post-mortem debugging of an exception that already happened.

# safety.py
import os
import pdb
import sys

os.environ["PYTHONBREAKPOINT"] = "0"
breakpoint()
print("breakpoint() ignored because PYTHONBREAKPOINT=0")


def divide(a, b):
    return a / b


try:
    divide(1, 0)
except ZeroDivisionError:
    tb = sys.exc_info()[2]
    frames = []
    while tb is not None:
        frames.append((tb.tb_frame.f_code.co_name, tb.tb_lineno))
        tb = tb.tb_next
    print("post-mortem frames:", frames)
    print("pdb.post_mortem is", callable(pdb.post_mortem))

breakpoint() ignored because PYTHONBREAKPOINT=0
post-mortem frames: [('<module>', 15), ('divide', 11)]
pdb.post_mortem is True

Try it yourself

Take a function you wrote earlier that has a loop, put a breakpoint() inside it, run the script, and practice p, n, s, l, w, and c until they are automatic. Then write a test that fails, run pytest --pdb, and inspect the values with p before quitting. Finally, set a breakpoint in VS Code on the same line, run with F5, and compare the variables pane with what p showed you.

Common mistakes

• Leaving breakpoint() in committed code. It halts the program wherever it runs; search for it before committing, and know PYTHONBREAKPOINT=0 for emergencies.
• Debugging without reproducing. If the bug appears one time in ten, get it to appear every time first, ideally as a test; the debugger cannot inspect what does not happen.
• Fixing the symptom at the breakpoint. Seeing a bad value is the start; find the line that produced it, often one frame up with u.

What's next

• breakpoint() opens pdb; p, n, s, c, l, w, u, d, and q cover most sessions, and any other input is evaluated as Python.
• pytest --pdb debugs at the point of failure; VS Code offers the same with a graphical view; PYTHONBREAKPOINT=0 disables stray breakpoints.
• Reproduce, narrow, inspect, fix the cause, keep the test.

You can see inside a running program. Next your programs reach outside themselves, to web APIs over HTTP, using httpx and a local test server so nothing depends on the internet.