Teaching an LLM (distillation + RLAIF)

The previous chapter’s agents learned from a reward signal in a game. This one turns the same machinery on a language model: a large teacher (an API model) transfers competence into a small student (Qwen2.5-1.5B in the playground) through the standard modern post-training recipe — supervised fine-tuning, then preference optimization against the teacher’s judgements. That second stage is reinforcement learning from AI feedback, and by the end of the chapter you will have derived, line by line, why the playground can do it with a classification loss (DPO) instead of running PPO.

Stage 0 — what a language model is optimizing

An autoregressive LM defines $\pi_\theta(y \mid x) = \prod_t \pi_\theta(y_t \mid x, y_{<t})$, trained by next-token cross-entropy. Post-training never changes this machinery — it changes which distribution the model is pulled toward. Keeping that frame prevents the two chronic confusions of this area: fine-tuning does not “add a database” (it shifts a distribution), and RLHF-style methods do not “train on answers” (they reshape relative preferences between answers the model can already produce).

Stage 1 — SFT: distillation as supervised learning

Collect prompts $x$ with teacher (or curated) responses $y$, and minimize

— cross-entropy on the target tokens only (the prompt is context, not target; the playground masks it accordingly). This is behavioral cloning, the same “copy the expert” move that imitation learning makes, with the same characteristic strengths and failure. It transfers style, format and procedure extremely efficiently — a few hundred examples visibly reshape how the student answers. It transfers facts and competence only to the extent the student’s base distribution can support them: token-forcing a 1.5B model through statements it has no representation for yields memorization at best and confident hallucination at worst. The honest framing for small-student distillation: SFT mostly selects and sharpens behaviours that the pretrained model already latently contains.

LoRA: why 3 GB of VRAM is enough

Full fine-tuning updates every weight matrix. Low-Rank Adaptation freezes the pretrained $W_0 \in \mathbb{R}^{d \times k}$ and learns only a low-rank correction:

For a 4096×4096 attention projection at $r = 16$: 16.8M frozen parameters, 131K trainable — 0.8% — and the optimizer states (Adam keeps two moments per parameter, the real memory hog) shrink by the same factor. $B$ is initialized to zero so training starts exactly at the base model; $\alpha/r$ decouples the update’s scale from the chosen rank. The bets LoRA makes are worth stating: that the change needed for a task is low-rank (empirically true for style/format adaptation — which is precisely what SFT transfers best; less true for injecting broad new knowledge), and — recall the factorization caveat from the very first Foundations lesson — the objective is non-convex in $(A, B)$ even where it was convex in $\Delta W$, which in practice bothers no one. Adapters also compose: keep one per skill and swap or merge them — the mechanism the University chapter builds its skill registry on.

Stage 2 — from RLHF to DPO, the actual derivation

SFT clones; it cannot express “this answer is better than that one.” Preference data can: for a prompt $x$, two responses with a judgement $y_w \succ y_l$ (in the playground the student samples both and the teacher judges — RLAIF: same mathematics as RLHF, an AI where the human panel sat).

The preference model. Bradley–Terry (1952) posits a latent reward:

The RLHF objective. Classic RLHF fits a reward model $r_\phi$ to preferences, then optimizes the policy with PPO (last chapter’s engine), leashed to a reference policy (the SFT model) by a KL penalty so it can’t wander into reward-hacked gibberish:

Step 1 — this objective has a closed-form optimum. For each $x$, maximizing $\mathbb{E}_\pi[r] - \beta\,\text{KL}$ over distributions is solved by the Gibbs distribution (complete the KL and normalize):

Step 2 — invert it. Solve for the reward that makes any given $\pi$ optimal:

Step 3 — substitute into Bradley–Terry. The intractable $Z(x)$ is the same for $y_w$ and $y_l$ under one prompt, so it cancels in the difference, and the preference probability becomes a function of the policy alone. Fitting it by maximum likelihood gives Direct Preference Optimization:

Read what happened: the reward model, the sampling loop and PPO disappeared — not approximated away, but derived away. Your language model is secretly its own reward model (the paper’s slogan, now earned), and preference optimization became logistic regression (lesson 3 of Supervised Learning) on a margin $m$ built from four log-probabilities — two forward passes each on policy and frozen reference.

The trade-offs, honestly: DPO is offline (it fits the pairs it was given; PPO-style online RL can explore beyond them), it can lower the absolute likelihood of both responses while widening their gap (a known quirk — the objective only sees differences), and its strength depends on pair quality — which in RLAIF means judge quality, a dependency the University chapter quantifies with a widget.

Stage 3 — did it work? Evaluation without self-deception

Post-training evaluation is where wishful thinking goes to thrive; the playground ships the standard battery, and each metric answers a different question:

Three disciplines carry over from the Supervised Learning evaluation lesson with extra force here. Held-out means held out — the eval prompts must never appear in SFT or preference data, and with a teacher generating both training data and judgements, contamination is one lazy prompt-template away. Judges are biased instruments — LLM judges systematically favour longer answers, confident tone, and the first position shown (hence the playground’s position-swapped pairs at temperature 0). Report the spread — SFT and DPO runs vary across seeds and data orderings; a single win-rate without repetition is an anecdote.

And the capability ceiling, stated plainly since students will hit it in the lab: a 1.5B student can be made noticeably more useful in format and focus with a few hundred good pairs, and it will still hallucinate at the edge of its knowledge — post-training reshapes the distribution the pretraining built; it does not extend its support. Fine-tuning also bakes in a timestamp: for knowledge that moves, the architecture answer is retrieval (the LLM track’s territory), not more epochs.

Exercises