15. Inside the model: interpretability

We can read every number in a transformer, yet understanding what it computes is an open research field. A few ideas have become standard vocabulary: the residual stream as a shared channel, attention heads split into “where to look” and “what to move”, induction heads as a mechanism behind in-context learning, and superposition as the reason single neurons are hard to read.

The residual stream view

Each block adds its output into the stream, so the final vector at a position is the embedding plus the sum of every attention head’s and MLP’s contribution. Each component reads the stream through a linear projection and writes back by addition. The stream is therefore a communication channel with many subspaces, and behaviour can be decomposed into additive contributions (Elhage et al., 2021, “A Mathematical Framework for Transformer Circuits”).

Logit lens: apply the final norm and the unembedding to the residual stream after any intermediate layer to see which token the model “would predict now”. Predictions often sharpen gradually across layers: the model refines a guess rather than computing it all at the end.

QK and OV circuits

Each attention head does two separable things:

• The QK circuit (W_Q and W_K together) decides where to attend: which source positions match the current one.
• The OV circuit (W_V and W_O together) decides what information moves from the attended position into the current position’s stream.

Many heads become interpretable once these are analysed separately: previous-token heads, bracket-matching heads, name-copying heads.

Induction heads

Text repeats: if “Harry Potter” appeared earlier, then after the next “Harry” the token “Potter” is likely. Induction heads implement the rule [A][B] … [A] → predict [B], using two heads in two layers:

1. A previous-token head writes into each position “the token before me was X”.
2. An induction head at the current token A searches for earlier positions whose previous token was A - that is where B sits - attends there, and copies B to the output.

They appear during training in a sudden phase change that coincides with a jump in in-context learning ability (Olsson et al., 2022). Fuzzier versions generalize beyond exact copying to pattern completion. This is the clearest mechanistic link between architecture and the in-context learning that makes prompting work.

MLPs and stored associations

MLPs behave somewhat like key–value memories: the first layer detects patterns in the stream (keys), and the second writes associated information (values). Causal tracing localized factual recall (“The Eiffel Tower is in” → “Paris”) to mid-layer MLPs at the subject’s last token, and showed facts can be edited there (ROME, Meng et al., 2022), though such edits generalize imperfectly.

Superposition and features

Ideally one neuron would mean one thing; in practice neurons are polysemantic, firing for unrelated concepts. The superposition hypothesis explains why. A model needs far more features than it has dimensions, and because most features are rarely active together, it packs them into nearly orthogonal directions and tolerates a little interference.

Sparse autoencoders (dictionary learning) try to undo this: train a wide, sparse layer to reconstruct activations so that each learned direction corresponds to one interpretable feature. Anthropic’s 2023–2024 work extracted millions of features from a production-scale model, including abstract ones, and showed that amplifying a feature changes behaviour predictably. Related is the linear representation hypothesis - many concepts are directions in activation space - which underlies steering vectors: add a direction to shift behaviour.

Methods and their pitfalls

Method	What it tells you	Pitfall
Attention pattern visualization	where heads look	attention weights are not explanations: values may be tiny, or later layers may discard the result
Linear probes	whether information is linearly decodable from activations	decodable ≠ used by the model
Activation patching / causal tracing	which components causally matter for a behaviour	results depend on the chosen corrupted baseline
Logit lens	intermediate predictions	can mislead in early layers; tuned variants help
Sparse autoencoders	candidate interpretable features	imperfect reconstruction; features split or merge as dictionary size changes

Tools to try: TransformerLens and nnsight for hooking into models; Anthropic’s Transformer Circuits thread for the research.

Why this matters for systems work

Interpretability is increasingly practical: debugging unexpected behaviour, detecting shortcut reasoning, monitoring for unsafe internal states, and steering behaviour without retraining. It also sharpens intuition. Once you see the residual stream as a shared bus, design choices like pre-norm, residual scaling and why some layers can be pruned start to make sense.

Questions you should now be able to answer

• Why is the final residual stream a sum of component contributions, and what does that enable?
• Describe the two-head induction circuit and why it needs two layers.
• What is the difference between a probe finding information and the model using it?
• Why does superposition make individual neurons polysemantic?
• How would you test whether a specific attention head is necessary for a behaviour?