Reasoning loops

The last chapter improved a model by editing its weights. This one improves answers without touching a single weight: sample more, critique, refine, vote — spending inference-time compute instead of training-time compute. These patterns went from prompt-engineering folklore to a central scaling axis of the field (OpenAI’s o-series and every “thinking mode” since are this idea, industrialized via RL — the chapter ends where that begins). The playground’s Reasoning Lab lets you wire the patterns yourself; this chapter supplies the theory of when they work, the math of when they fail, and the accounting that decides if they’re worth it.

The frame: an inference-time policy

A reasoning loop is a tiny program whose primitive operation is an LLM call: generators propose, critics score, refiners rewrite, aggregators select. In RL vocabulary the graph is a policy over calls — state = the working answer and its critiques; actions = which node runs next; the critic’s verdict is a learned (and noisy) reward signal steering control flow. That framing earns the chapter its place in this track, and it makes one thing obvious immediately: everything depends on the quality of that reward signal. Hold the thought; it is the chapter’s spine.

Self-consistency: the exact mathematics of majority voting

The simplest pattern needs no critic at all: sample $k$ independent chains of thought (temperature > 0), take the majority answer. If each sample is correct independently with probability $p$, the vote’s accuracy is a binomial tail:

— Condorcet’s jury theorem, 1785, now running on GPUs. Its two regimes are the whole strategy in one formula, and you should feel them:

The economics follow directly: compute scales linearly in $k$ while the curve flattens exponentially, so self-consistency is spectacular for the first few samples on tasks where $p$ is middling, and pure waste once the curve saturates — or worse than waste below the line. Diagnosis before dosage: estimate $p$ first.

Critique and refine: Self-Refine, Reflexion, and the verifier gap

The second family adds the critic: generate → critique → refine, looping until the critic passes the answer or a budget runs out (exactly the loop you can wire in the Lab, with the critic’s ✓/✗ routing the graph). Two canonical instances frame the design space: Self-Refine (the same model critiques its own output within one episode) and Reflexion (critiques persist as memory across episodes — the agent accumulates lessons, an inference-time cousin of learning).

When does the loop actually converge to better answers? The honest empirical record reduces to one principle — the generator–verifier gap:

A critique loop helps exactly when verifying is easier than generating — when the critic’s effective accuracy on this task exceeds the generator’s.

That’s why the wins concentrate where verification has traction: code (run the tests — a critic with $p \approx 1$), math with checkable steps, format and constraint compliance, retrieval-grounded factuality. And why the failures concentrate where verification is as hard as generation: the same model re-reading its own open-ended reasoning mostly re-likes it (its errors and its judgements are correlated — the independence failure again, now within one model), so intrinsic self-correction on open-ended tasks reliably underdelivers (Huang et al.). The practical corollaries, each one buildable in the Lab: make critics external (tools, tests, retrieval) where possible; make them specialized and decomposed (a relevance critic and a correctness critic beat one vague “is this good?” prompt); and bound the loop — refinement oscillates or drifts stylistically after a few iterations, so cap it and keep the best-so-far rather than the last.

The accounting: loop vs bigger model vs fine-tune

Inference-time compute is a purchase, and it competes with two other line items. The decision discipline:

Rules of thumb that survive contact: measure the cost per quality point (tokens spent ÷ accuracy gained) before shipping a loop; a loop that quadruples cost for two points loses to a one-off fine-tune if the volume is high — and the hybrid (distil the loop’s outputs into the weights via SFT, last chapter’s machinery) is often the endgame: run the expensive loop offline, train the cheap model on its wins.

Where this ends: training the loop into the weights

Everything above treats the model as fixed. The current frontier closes the circle with this track’s RL: generate many chains, reward the ones that reach verified answers, and update the policy — outcome- and process-reward RL (PPO/GRPO-style, the previous chapters’ engines) that moves the reasoning into the weights, so the deployed model spends its inference compute more effectively. Reasoning models are, literally, reasoning loops distilled by reinforcement learning — which is why this chapter sits in an RL track, and why the skills compose: the graph you wire by hand in the Lab today is the training curriculum of the reasoning models you’ll study in the LLM track.

Exercises