Post-training: from text predictor to assistant · Transformers

A pretrained base model continues documents; it doesn’t answer, follow instructions or decline anything. Post-training turns it into an assistant in stages: supervised fine-tuning on example conversations, then optimization against preferences or verifiable rewards. It changes behaviour far more than knowledge, and many quirks you’ll meet in production - formatting habits, verbosity, refusals, sycophancy - come from it.

What a base model does

Prompt a base model with “What is the capital of France?” and it may write more quiz questions, because that’s how such lines continue on the web. It can still be steered with document-style prompts (“Q: … A:”), but it has no notion of a user or of its own role.

Stage 1: supervised fine-tuning (SFT)

Train on (prompt, ideal response) pairs with the same next-token cross-entropy, but count the loss only on response tokens (loss masking), so the model learns to produce answers rather than imitate user messages. Data comes from human-written demonstrations, filtered model outputs and synthetic data from stronger models. Quality beats quantity: a few thousand excellent examples go a long way, because SFT mostly teaches format and behaviour on top of what pretraining already built.

Chat templates

A conversation is serialized into one token sequence, with special tokens marking roles and turn boundaries. An illustrative ChatML-style example:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is RoPE?<|im_end|>
<|im_start|>assistant

The model learned these markers during post-training. Using the wrong template, or letting user-supplied text contain these special tokens, degrades or subverts behaviour. When serving open models yourself, always apply the model’s own template (Hugging Face tokenizers ship it as chat_template).

Stage 2: learning from preferences - RLHF

People find it easier to compare two answers than to write the ideal one. RLHF, as in InstructGPT (2022), builds on that:

Collect comparisons. For a prompt x, sample two responses; a person marks the preferred y_w over the rejected y_l.
Train a reward model r(x, y) with the Bradley–Terry loss, so preferred answers score higher:

\mathcal{L}_{RM} = -\log \sigma\big(r(x, y_w) - r(x, y_l)\big)

Optimize the policy (the LLM) with reinforcement learning, typically PPO, to maximize reward while staying close to the SFT model:

\max_{\pi}\; \mathbb{E}_{y \sim \pi(\cdot \mid x)}\big[r(x, y)\big] - \beta\, \mathrm{KL}\big(\pi(\cdot \mid x)\,\|\,\pi_{ref}(\cdot \mid x)\big)

The KL penalty matters. The reward model is an imperfect proxy, and a policy pushed too hard finds outputs that score high without being good (reward hacking): overly long, flattering or confident answers, for example.

DPO: preferences without the RL loop

Direct Preference Optimization (Rafailov et al., 2023) notes that the optimal policy of the KL-regularized objective defines an implicit reward, β·log(π/π_ref). Substituting it into the Bradley–Terry loss gives a loss computed directly on preference pairs - no reward model, no sampling, no PPO:

\mathcal{L}_{DPO} = -\log \sigma\!\left( \beta \log \frac{\pi(y_w \mid x)}{\pi_{ref}(y_w \mid x)} - \beta \log \frac{\pi(y_l \mid x)}{\pi_{ref}(y_l \mid x)} \right)

It’s stable and simple, so it’s popular for open models. Relatives (IPO, KTO, ORPO, SimPO) change the loss or the data format. Online RL methods still tend to win at the frontier because they learn from the model’s own fresh samples.

AI feedback and written principles

Human labels are slow and expensive. RLAIF uses a model to produce preference labels. Constitutional AI (Anthropic, 2022) has the model critique and revise its outputs against a written set of principles, and uses AI-generated preferences to train harmlessness. Both scale labelling and make the trained values more explicit.

RL with verifiable rewards and reasoning models

For math, code and other checkable tasks, the reward can come from a verifier: does the final answer match, do the unit tests pass? Trained this way at scale, models learn to produce long chains of thought before answering - exploring, checking, backtracking - and accuracy rises with the number of tokens they may think for. This adds a new scaling axis: test-time compute.

GRPO (from DeepSeekMath, used for DeepSeek-R1) is a common algorithm. It samples a group of answers per prompt, scores each, and uses each answer’s reward relative to the group average as its advantage, so no separate value network is needed. Outcome rewards judge only the final answer; process reward models judge the intermediate steps.

Side effects to watch for

Reward hacking: exploiting flaws in the reward, such as tests that can be gamed.
Sycophancy: agreeing with the user because raters liked agreement.
Verbosity and formatting bias: longer, heavily formatted answers often win comparisons.
Calibration drift: base models are often reasonably calibrated; preference tuning can make them overconfident.
Over-refusal vs under-refusal: safety training trades one against the other.

Questions you should now be able to answer

Why compute SFT loss only on the assistant’s tokens?
What does the KL term in RLHF protect against, and what happens if β is too small?
Where does DPO’s “implicit reward” come from?
Why do verifiable rewards work well for math and code but poorly for essays?
A deployed model agrees with users’ wrong claims. Which training stage most likely caused it, and how would you test that?