Advanced error handling

What you'll learn

• Chain exceptions with raise ... from so the cause is never lost
• Inspect and format tracebacks with the traceback module
• Emit and control warnings for problems that are not errors yet
• Write retry logic with backoff that gives up after a bounded number of attempts

Concept

The Beginner exceptions lesson taught you to catch, raise, and design your own types. Production code needs three more habits. First, when you catch a low-level error and raise a higher-level one, keep the original: raise PaymentError("charge failed") from err. The traceback then shows both, with "The above exception was the direct cause of the following exception", so whoever reads the log sees the ConnectionError that caused the PaymentError. Without from, Python still links them as "during handling", but from says explicitly that one caused the other, and from None deliberately hides an internal cause when it would only confuse the caller.

Second, tracebacks are data, not only text on a terminal. The traceback module formats them, traceback.format_exc() returns the current one as a string for a log or an error report, and traceback.extract_tb() gives the frames as objects you can inspect. The __cause__, __context__, and __traceback__ attributes on any exception hold the chain and the frames. logging.exception uses exactly this machinery, and so should any error reporter you write.

Third, not every problem is an error. A deprecated parameter, a config value that will stop working next version, a fallback that silently degrades a result: these should be reported without stopping the program. The warnings module does this. warnings.warn("use timeout_s instead", DeprecationWarning) prints once per location by default. Filters let a test suite turn every warning into an error so deprecations are caught early, and let a script silence a noisy library warning it has assessed. add_note on an exception attaches extra context, such as which record was being processed, as it travels up.

Retries are where error handling meets time. Network calls fail transiently; the right response is to try again, wait a little longer each time, and give up after a bounded number of attempts with a clear error that chains the last failure. Retry only errors that are plausibly transient, never a ValueError from bad input, and log each attempt so the retries are visible. A retry loop without a limit is an outage that never reports itself.

Code

Chaining with from, and the difference from None makes. The first traceback shows cause and effect; the second shows only the translated error.

# chaining.py
class PaymentError(Exception):
    pass


def charge(card):
    try:
        raise ConnectionError("gateway timeout")
    except ConnectionError as err:
        raise PaymentError(f"charge failed for {card}") from err


def lookup(user_id):
    try:
        return {"1": "ada"}[user_id]
    except KeyError:
        raise ValueError(f"no such user: {user_id}") from None


try:
    lookup("9")
except ValueError as err:
    print(repr(err), "| cause:", err.__cause__, "| context suppressed:", err.__suppress_context__)

charge("4242")

ValueError('no such user: 9') | cause: None | context suppressed: True
Traceback (most recent call last):
  File "chaining.py", line 7, in charge
    raise ConnectionError("gateway timeout")
ConnectionError: gateway timeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "chaining.py", line 24, in <module>
    charge("4242")
    ~~~~~~^^^^^^^^
  File "chaining.py", line 9, in charge
    raise PaymentError(f"charge failed for {card}") from err
PaymentError: charge failed for 4242

Tracebacks as data: capture the current one as a string, walk its frames, and attach a note to an exception on its way up.

# traceback_tools.py
import traceback


def parse_row(row):
    return int(row["amount"])


def process(rows):
    for i, row in enumerate(rows):
        try:
            parse_row(row)
        except ValueError as err:
            err.add_note(f"while processing row {i}: {row}")
            raise


try:
    process([{"amount": "10"}, {"amount": "ten"}])
except ValueError as err:
    text = traceback.format_exc()
    print("lines in traceback:", len(text.splitlines()))
    frames = traceback.extract_tb(err.__traceback__)
    print([(f.name, f.lineno) for f in frames])
    print("notes:", err.__notes__)
    print(text.splitlines()[-2:])

lines in traceback: 11
[('<module>', 18), ('process', 11), ('parse_row', 5)]
notes: ["while processing row 1: {'amount': 'ten'}"]
["ValueError: invalid literal for int() with base 10: 'ten'", "while processing row 1: {'amount': 'ten'}"]

Warnings: emitted once, turned into errors under a filter, and silenced when assessed. The category tells tools and readers what kind of problem it is.

# warn.py
import warnings


def connect(host, timeout=None, timeout_s=5):
    if timeout is not None:
        warnings.warn("'timeout' is deprecated, use 'timeout_s'", DeprecationWarning, stacklevel=2)
        timeout_s = timeout
    return f"{host} ({timeout_s}s)"


with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    connect("db", timeout=3)
    connect("db", timeout=3)
print(len(caught), caught[0].category.__name__, str(caught[0].message))

with warnings.catch_warnings():
    warnings.simplefilter("error", DeprecationWarning)
    try:
        connect("db", timeout=3)
    except DeprecationWarning as err:
        print("as error:", err)

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    print(connect("db", timeout=3), "(warning silenced)")

2 DeprecationWarning 'timeout' is deprecated, use 'timeout_s'
as error: 'timeout' is deprecated, use 'timeout_s'
db (3s) (warning silenced)

A retry helper with exponential backoff, bounded attempts, and a chained final error. Non-transient errors are not retried. The sleep is injected so the demo runs instantly.

# retry.py
import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
log = logging.getLogger("retry")


class RetryError(Exception):
    pass


def retry(func, attempts=3, base_delay=0.5, retry_on=(ConnectionError, TimeoutError), sleep=None):
    sleep = sleep or (lambda s: None)
    delay = base_delay
    last = None
    for attempt in range(1, attempts + 1):
        try:
            return func()
        except retry_on as err:
            last = err
            log.warning("attempt %d/%d failed: %s; waiting %.1fs", attempt, attempts, err, delay)
            sleep(delay)
            delay *= 2
    raise RetryError(f"gave up after {attempts} attempts") from last


state = {"calls": 0}


def flaky():
    state["calls"] += 1
    if state["calls"] < 3:
        raise ConnectionError("reset by peer")
    return "ok"


print(retry(flaky))

try:
    retry(lambda: int("x"))
except ValueError as err:
    print("not retried:", err)

retry(lambda: (_ for _ in ()).throw(TimeoutError("slow")), attempts=2)

WARNING: attempt 1/3 failed: reset by peer; waiting 0.5s
WARNING: attempt 2/3 failed: reset by peer; waiting 1.0s
ok
not retried: invalid literal for int() with base 10: 'x'
WARNING: attempt 1/2 failed: slow; waiting 0.5s
WARNING: attempt 2/2 failed: slow; waiting 1.0s
Traceback (most recent call last):
  File "retry.py", line 17, in retry
    return func()
  File "retry.py", line 43, in <lambda>
    retry(lambda: (_ for _ in ()).throw(TimeoutError("slow")), attempts=2)
                  ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "retry.py", line 43, in <genexpr>
    retry(lambda: (_ for _ in ()).throw(TimeoutError("slow")), attempts=2)
TimeoutError: slow

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "retry.py", line 43, in <module>
    retry(lambda: (_ for _ in ()).throw(TimeoutError("slow")), attempts=2)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "retry.py", line 23, in retry
    raise RetryError(f"gave up after {attempts} attempts") from last
RetryError: gave up after 2 attempts

Try it yourself

Create importer.py that reads records from a list and raises a custom ImportError-style exception of your own, RecordError, chained from the underlying KeyError or ValueError, with a note naming the record index. Catch it at the top, log it with logging.exception, and print the cause chain by following __cause__. Then wrap a flaky function with the retry helper, once with a failure that recovers and once that exhausts the attempts, and confirm a ValueError is not retried.

Common mistakes

• Catching and re-raising without from. The original cause is downgraded to "during handling" context, and readers cannot tell whether it was the reason.
• Unbounded retries. A loop with no attempt limit turns a transient failure into a silent hang; always cap attempts and raise a clear error at the end.
• Retrying everything. A ValueError or TypeError will fail identically every time; retry only errors that can plausibly resolve themselves.

What's next

• raise X from err chains cause to effect for readers of the traceback; from None hides internal causes on purpose.
• traceback turns tracebacks into strings and frames; add_note attaches context; warnings reports problems that are not yet errors.
• Retry only transient errors, with growing delays, a hard attempt limit, and a final chained error.

Your programs fail well. Next you will make sure they work at all, systematically, with pytest.