12. Concurrency and parallelism

First classify the work: I/O-bound work spends its time waiting (network, disk, LLM APIs) and needs concurrency; CPU-bound work spends it computing and needs parallelism. The right tool follows from that classification.

The GIL

• The Global Interpreter Lock lets only one thread execute Python bytecode at a time in a standard CPython process. It cheaply protects interpreter internals such as reference counts.
• Threads still help with I/O, because a thread waiting on a socket releases the GIL.
• C extensions like NumPy and PyTorch release the GIL during heavy computation, so threads calling into them can overlap.
• Python 3.13 added an experimental free-threaded (no-GIL) build, and 3.14 made it officially supported but optional. The default build still has the GIL and library support is still catching up, so check your stack before relying on it.

Threads

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=16) as pool:
    pages = list(pool.map(download, urls))

Threads suit blocking I/O with synchronous libraries. The risk is shared mutable state: two threads updating the same object can interleave and corrupt it. Protect shared state with threading.Lock, or pass data through a queue.Queue.

Processes

from concurrent.futures import ProcessPoolExecutor

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        features = list(pool.map(extract_features, files, chunksize=64))

• Each process has its own interpreter and its own GIL, which gives real CPU parallelism.
• The costs: process startup, plus every argument and result is pickled to cross the process boundary. Large objects mean large overhead.
• Start methods matter. “fork” copies the parent process and is unsafe once threads or CUDA are initialized; “spawn” starts fresh and re-imports your module, which is why the __main__ guard is required.
• Never fork after initializing CUDA. Use “spawn” for GPU multiprocessing.
• PyTorch’s DataLoader with num_workers > 0 runs worker processes that load and augment data while the GPU trains.

asyncio: one thread, many waiting tasks

An event loop runs coroutines. A coroutine runs until it reaches an await on something not yet ready, then hands control back so another coroutine can run. There is no parallelism; the program simply never sits idle while waiting.

import asyncio
import httpx

async def classify(client: httpx.AsyncClient, text: str, sem: asyncio.Semaphore) -> dict:
    async with sem:                               # at most N requests in flight
        r = await client.post("/classify", json={"text": text}, timeout=30)
        r.raise_for_status()
        return r.json()

async def main(texts: list[str]) -> list:
    sem = asyncio.Semaphore(10)
    async with httpx.AsyncClient(base_url="https://api.example.com") as client:
        return await asyncio.gather(
            *(classify(client, t, sem) for t in texts),
            return_exceptions=True,               # one failure doesn't lose the rest
        )

results = asyncio.run(main(texts))

If each call takes about one second, 1,000 calls take about 1,000 seconds sequentially and about 100 seconds with 10 in flight, on a single thread.

The rules of async code:

• Calling an async def function returns a coroutine object that does nothing until it is awaited or scheduled as a task.
• Never call blocking code inside a coroutine: time.sleep, requests, sync database drivers or heavy CPU work freeze the whole event loop. Use asyncio.sleep, async clients, or await asyncio.to_thread(blocking_fn, ...).
• Fan out with asyncio.gather, or asyncio.TaskGroup (3.11+), which cancels the remaining tasks if one fails. Set deadlines with asyncio.timeout.
• Async spreads upward: a function that awaits must itself be async. Decide where the async boundary sits, for example at FastAPI endpoints.
• Async SDKs stream tokens through async iterators: async for chunk in stream:.

Questions to ask:

• Is this work I/O-bound or CPU-bound?
• How many requests are in flight, and what is the provider’s rate limit?
• Is anything blocking the event loop?
• What gets pickled and sent to worker processes, and how big is it?