Processes and executors

What you'll learn

• Run CPU-bound work in parallel with ProcessPoolExecutor and measure the speedup
• Understand what processes cost: startup, pickling, and no shared memory
• Choose between threads, processes, and 3.14's concurrent.interpreters
• Handle results, exceptions, and timeouts from a pool the same way for all three

Concept

Processes are the answer the GIL forces for computation. Each worker process has its own interpreter and its own GIL, so four processes on four cores compute four times as fast as one, for work that is genuinely CPU-bound. concurrent.futures.ProcessPoolExecutor has the same interface as the thread pool, map, submit, as_completed, result, so switching between them is one word. The concurrent.futures module exists so that the choice of executor is a configuration, not a rewrite.

The costs are real and specific. Starting a process takes tens of milliseconds and imports your modules again, so a pool is worth it for work that takes longer than that. Arguments and results travel by pickling, so functions must be importable at module level, not lambdas or closures, and large data crosses the boundary slowly; pass file paths or chunks rather than whole datasets. Nothing is shared: a global updated in a worker is invisible to the parent. Design process work as pure functions from input to output and let the pool handle transport.

Python 3.14 adds a third option: concurrent.interpreters, multiple interpreters in one process, each with its own GIL. They start faster than processes and share the process's memory space while still isolating Python objects, and InterpreterPoolExecutor gives them the same executor interface. It is new, so libraries are still adapting, but it points where CPU parallelism in Python is going. On the free-threaded build, plain threads compute in parallel too, which makes the question "which executor" increasingly one of isolation rather than speed.

Whatever executor you use, handle the three things that go wrong. A worker that raises stores the exception in its future and re-raises it when you call result(), so wrap that call. A worker that hangs is bounded with result(timeout=) or wait(timeout=). And a pool that is not shut down leaves workers alive, so always use the executor as a context manager. Test the function you submit as an ordinary function first; the pool adds transport, not logic.

Code

The same CPU-bound function from the threads lesson, now in a process pool. Threads gave no speedup; processes do, in proportion to the cores actually free. The checks below prove where the work ran rather than how long it took, because a stopwatch is not portable: os.cpu_count() reports the machine's cores and ignores a container's CPU limit, so the same code that runs four times faster on your laptop can run no faster inside a build container that is allowed one core.

# cpu_parallel.py
import os
import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor


def count(n):
    total = 0
    for i in range(n):
        total += i
    return total, os.getpid()


if __name__ == "__main__":
    N, JOBS = 8_000_000, 4

    start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=JOBS) as pool:
        threaded = list(pool.map(count, [N] * JOBS))
    thread_seconds = time.perf_counter() - start

    start = time.perf_counter()
    with ProcessPoolExecutor(max_workers=JOBS) as pool:
        parallel = list(pool.map(count, [N] * JOBS))
    process_seconds = time.perf_counter() - start

    print("same answers:", {t for t, _ in threaded} == {t for t, _ in parallel})
    print("threads all ran in this process:", {p for _, p in threaded} == {os.getpid()})
    print("processes ran somewhere else:", os.getpid() not in {p for _, p in parallel})
    print("pool overhead stayed small:", process_seconds < thread_seconds * 2)

same answers: True
threads all ran in this process: True
processes ran somewhere else: True
pool overhead stayed small: True

The pickling boundary: a module-level function works, a lambda does not, and a worker's global change never reaches the parent.

# boundaries.py
from concurrent.futures import ProcessPoolExecutor

COUNTER = 0


def square(x):
    return x * x


def bump_global():
    global COUNTER
    COUNTER += 100
    return COUNTER


if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=2) as pool:
        print(list(pool.map(square, range(5))))
        try:
            pool.submit(lambda x: x + 1, 1).result()
        except Exception as err:
            print("lambda:", type(err).__name__)
        worker_value = pool.submit(bump_global).result()
    print("worker saw", worker_value, "| parent still has", COUNTER)

[0, 1, 4, 9, 16]
lambda: PicklingError
worker saw 100 | parent still has 0

Exceptions and timeouts through futures. A failing worker's error surfaces at result(), a slow worker is bounded, and the others complete normally.

# futures.py
import time
from concurrent.futures import ProcessPoolExecutor, TimeoutError, as_completed


def work(job):
    name, seconds = job
    if name == "bad":
        raise ValueError("bad input")
    time.sleep(seconds)
    return f"{name} done"


if __name__ == "__main__":
    jobs = [("fast", 0.1), ("bad", 0.1), ("slow", 3.0), ("medium", 0.3)]
    with ProcessPoolExecutor(max_workers=4) as pool:
        futures = {pool.submit(work, job): job[0] for job in jobs}
        for future in as_completed(futures, timeout=1.5):
            name = futures[future]
            try:
                print(name, "->", future.result())
            except ValueError as err:
                print(name, "-> raised", err)
            if name == "medium":
                break
        slow = next(f for f, n in futures.items() if n == "slow")
        try:
            slow.result(timeout=0.1)
        except TimeoutError:
            print("slow -> still running after the timeout; cancelling:", slow.cancel() or "too late, it is running")

bad -> raised bad input
fast -> fast done
medium -> medium done
slow -> still running after the timeout; cancelling: too late, it is running

Multiple interpreters in 3.14: the same executor interface with a lighter unit of parallelism. The function must be importable, as with processes, but the work stays inside one process, which is what makes it lighter.

# interpreters.py
import os
import sys

if __name__ == "__main__":
    try:
        from concurrent.futures import InterpreterPoolExecutor
    except ImportError:
        print("InterpreterPoolExecutor needs Python 3.14+; running", sys.version.split()[0])
        raise SystemExit

    def expected(n):
        total = 0
        for i in range(n):
            total += i
        return total

    from cpu_parallel import count as importable_count

    with InterpreterPoolExecutor(max_workers=4) as pool:
        results = list(pool.map(importable_count, [2_000_000] * 4))

    print(len(results), all(total == expected(2_000_000) for total, _ in results))
    print("all in this one process, unlike a process pool:", {p for _, p in results} == {os.getpid()})

4 True
all in this one process, unlike a process pool: True

Try it yourself

Write hash_files.py that creates twenty files of a few megabytes each in tmp/, then computes each file's SHA-256 with hashlib three ways: sequentially, in a ThreadPoolExecutor, and in a ProcessPoolExecutor. Print the three times and explain in a comment why the thread version is also fast here (hint: hashlib releases the GIL). Then make one worker raise on a specific file and confirm the error surfaces at result() without stopping the others.

Common mistakes

• Missing the if __name__ == "__main__": guard. With the default start methods (spawn on macOS and Windows, forkserver on Linux since 3.14) the pool re-imports your script to start workers; without the guard, each worker starts another pool.
• Submitting lambdas or closures. They cannot be pickled; use module-level functions.
• Sending big data through the pool. Pickling a large list per task can cost more than the work; send paths or chunks and let workers load their own.

What's next

• ProcessPoolExecutor runs CPU-bound Python in parallel; the interface matches the thread pool, so the executor is a one-word choice.
• Processes cost startup and pickling and share nothing; write importable, pure worker functions and pass small inputs.
• Futures carry results or exceptions; bound waits with timeouts; 3.14's InterpreterPoolExecutor is the lighter alternative.

You can compute in parallel. Next you will handle thousands of concurrent waits in a single thread, with async and await.