17. Tooling, debugging and testing

Fluency is mostly fast feedback: run small pieces interactively, step through failures in a debugger, and lock behavior in with tests. Code written by AI assistants needs the last step most of all.

Interactive work

• REPL: python works; IPython (ipython) is better, with tab completion, obj? for help, %timeit, and %debug after an exception.
• Jupyter: great for exploration and plots. Its trap is hidden state: cells run out of order, or only work after another cell ran. Use “Restart and Run All” before trusting a result.
• Notebooks plus modules: move stable code into .py modules and import it, with %load_ext autoreload and %autoreload 2 so edits are picked up.
• Introspection: type, dir, help, vars, inspect.getsource and type(obj).__mro__ answer most “what is this thing?” questions.

Debugging

• breakpoint() drops you into pdb at that line. The core commands: n (next line), s (step into), c (continue), p expr (print), pp (pretty-print), l (list code), w (where in the stack), u and d (move up or down the stack), q (quit).
• After a crash, python -m pdb script.py or %debug in IPython opens the frame that failed, with every local variable available.
• IDE debuggers (VS Code, PyCharm) are the same tool with a GUI. Learn conditional breakpoints.
• For tensors, print x.shape, x.dtype, x.device before printing values. torch.autograd.set_detect_anomaly(True) finds the operation that produced a NaN; it is slow, so use it only while debugging.
• The method: reproduce with the smallest input, form one hypothesis, change one thing, fix, then add a test that would have caught it.

Code quality tools

• ruff: linter and formatter in one fast tool. ruff check --fix and ruff format.
• mypy or pyright: static type checking (Part 10).
• pre-commit: runs these automatically on every commit.
• Configure all of them in pyproject.toml.

Testing with pytest

# tests/test_chunking.py
import pytest
from my_project.text import chunk

def test_chunks_respect_size():
    parts = chunk("a" * 2500, size=1000, overlap=100)
    assert all(len(p) <= 1000 for p in parts)

@pytest.mark.parametrize("size", [1, 10, 1000])
def test_no_empty_chunks(size):
    assert all(chunk("hello world", size=size, overlap=0))

• Plain assert statements: pytest rewrites them to show the actual values when they fail.
• Fixtures (@pytest.fixture) provide setup such as temporary directories (tmp_path), fake clients and tiny models.
• parametrize runs one test across many inputs; monkeypatch sets environment variables or replaces functions.
• Test pure logic heavily: parsing, chunking, prompt building, validation and tool routing.

Testing code that calls LLMs

• Unit tests never call the real API. Inject the client through a Protocol (Part 7) and pass a fake that returns canned responses: fast, free and deterministic.
• Record real responses once and replay them in tests for realistic fixtures.
• Test the failure paths on purpose: timeouts, 429s, malformed JSON and tool errors.
• For models, test shapes, check that every parameter receives a gradient, and run a single-batch overfit test. A working training loop drives the loss near zero on one repeated batch; if it cannot, the loop is broken.

Evals: tests for probabilistic behavior

• An eval is a fixed set of inputs with expected properties: an exact answer, required facts, a valid schema, the right tool chosen.
• Scorers range from exact match and programmatic checks to an LLM judge with a rubric. Validate any LLM judge against human labels first.
• Run evals on every prompt or model change, track scores over time, and compare score distributions rather than anecdotes.
• Start with 20 to 50 real examples and grow the set from production failures.

Questions to ask:

• How would I find out if this broke?
• Does this test hit the network?
• Did I restart the notebook and run everything top to bottom?