11. The performance model

Python is slow per operation, yet AI workloads in Python are fast because the heavy math runs in compiled kernels. Performance work is mostly moving loops out of the interpreter and keeping the hardware busy.

Why pure Python is slow

• Interpretation: every bytecode instruction passes through the interpreter loop.
• Dynamic dispatch: a + b looks up both types and their addition function at runtime, every time.
• Boxing: each number is a heap object with a reference count and a type pointer, not a raw value.
• Pointer chasing: lists hold pointers to objects scattered in memory, so the CPU cache keeps missing.
• One thread at a time: in the default build, only one thread runs Python bytecode at once (the GIL, Part 12).

The result is a gap of roughly two to three orders of magnitude between a pure Python loop and the same math in a vectorized library, and more on a GPU.

Why AI in Python is fast anyway

Python is the control plane; compiled libraries are the data plane.

# Pure Python: one interpreter round trip per element
total = 0.0
for i in range(len(a)):
    total += a[i] * b[i]

# One call: the loop runs in compiled code (SIMD, BLAS, or a GPU kernel)
total = a @ b

Same math, wildly different speed. Five rules follow from it:

1. Vectorize: express work as whole-array operations (Part 13).
2. Batch: many small calls cost more than one big call. This holds for GPU kernels, embedding APIs and database writes alike.
3. Keep per-element Python out of hot paths: .item(), .tolist(), .cpu() and Python loops over tensor elements each cross the boundary, and on a GPU they force a synchronization.
4. Keep data on the device: CPU-to-GPU copies are slow compared with GPU compute.
5. Choose the dtype deliberately: float32 or bfloat16 instead of float64 halves memory traffic or better.

Measure, don’t guess

• time.perf_counter() measures wall-clock spans.
• timeit, or %timeit in IPython, runs micro-benchmarks properly.
• cProfile shows which functions take the time: python -m cProfile -o out.prof script.py, then view it with snakeviz.
• py-spy samples a running process without code changes. It is the tool for “why is this job stuck?”
• line_profiler gives per-line timings inside one function.
• tracemalloc and memray track Python memory use.
• torch.profiler records a CPU and GPU timeline down to individual kernels.

GPU caveat: CUDA calls are asynchronous. Timing GPU code without torch.cuda.synchronize() measures only the kernel launch, not the work.

Wins, in order of typical payoff

1. Fix the algorithm: O(n²) to O(n log n), or list membership to set membership.
2. Move loops into libraries: NumPy, PyTorch, polars, pandas.
3. Batch I/O and API calls, and use concurrency for the waiting (Part 12).
4. Cache repeated work: functools.cache in memory, or an on-disk cache for embeddings and LLM responses.
5. Compile: torch.compile for models; Numba, Cython or Rust (via PyO3) for custom numeric loops.

Do not optimize code that does not appear in the profile.

Questions to ask:

• Where does the time go according to the profiler, not to intuition?
• Is there a Python loop over elements that could be one array operation?
• Is the GPU waiting on Python, on data loading, or on transfers?