RLHF and preference optimization

How a model that merely predicts text becomes one that’s helpful, honest and follows instructions. The field moved substantially between 2022 and 2026, and describing only classic PPO-based RLHF dates you.

Why preferences rather than labels

SFT teaches the model to imitate good responses. But for open-ended output there’s often no single correct answer, and it’s far easier for a human to say “A is better than B” than to write the ideal response.

Preference data is cheaper to collect, more consistent between annotators, and captures qualities — tone, helpfulness, appropriate refusal — that are hard to specify as a target string.

Classic RLHF (PPO)

The 2022 pipeline, three stages:

1. SFT — supervised fine-tuning on demonstrations.
2. Reward model — train a model to score responses, fit to human preference pairs.
3. PPO — optimise the policy against the reward model, with a KL penalty against the SFT model.

objective = E[reward(response)] - beta * KL(policy || reference)

The KL penalty is the load-bearing part. Without it the policy drifts to whatever maximises the reward model, which quickly means degenerate text that exploits the reward model’s blind spots — reward hacking. The KL term keeps it near a model that produces sensible language. See ../00_math_foundations/04_information_theory.md.

Why it fell out of favour: you must hold four models in memory (policy, reference, reward, value critic), the loop is unstable and hyperparameter-sensitive, and the learned reward model is itself a proxy that can be gamed.

DPO

Direct Preference Optimization removes the RL loop entirely. The insight is that the optimal policy under a KL-constrained reward objective has a closed form, so you can optimise directly on preference pairs with a simple classification-style loss:

# Conceptually: raise the log-prob of the chosen response relative to the rejected,
# measured against the frozen reference model.
loss = -log_sigmoid(beta * (
    (logp_policy_chosen - logp_ref_chosen) -
    (logp_policy_rejected - logp_ref_rejected)
))

No reward model, no sampling loop, no critic. Two models in memory instead of four, and it trains like ordinary supervised learning.

The trade-off: DPO is offline. It learns from a fixed set of preference pairs, so it can’t explore responses outside that data. On-policy methods generate fresh responses during training and can discover better ones.

Variants worth naming: IPO (addresses DPO’s tendency to overfit deterministic preferences), KTO (learns from single good/bad labels rather than pairs, so data is cheaper), ORPO (folds preference optimisation into SFT, skipping a stage).

GRPO

Group Relative Policy Optimization — the method behind much of the 2025-2026 reasoning work, and the one to be able to explain.

PPO needs a value critic to estimate the expected reward baseline. GRPO removes it: sample a group of responses to the same prompt, and use the group’s mean reward as the baseline. Each response’s advantage is its reward relative to its siblings.

advantage_i = (reward_i - mean(group_rewards)) / std(group_rewards)

No critic model, so roughly half the memory and compute of PPO, while staying on-policy — it generates fresh responses each step, unlike DPO.

That combination — on-policy exploration without the critic’s cost — is why the field converged on it for reasoning training.

DAPO refines GRPO for long chain-of-thought training, where it becomes unstable: asymmetric (“clip-higher”) clipping to preserve exploration, dynamic sampling that discards groups where every response is right or every one is wrong (zero gradient signal, wasted compute), token-level rather than sequence-level loss, and shaped rewards for overlong generations.

RLVR

Reinforcement Learning from Verifiable Rewards: replace the learned reward model with an automatic checker.

reward = 1.0 if run_tests(candidate) else 0.0

Where correctness is machine-checkable — maths with known answers, code with tests, formal logic — this removes the human labelling bottleneck and the reward-hacking failure mode simultaneously, because there’s no proxy to exploit.

It’s the core of how reasoning models are trained. The limitation is scope: it needs a cheap, reliable verifier, so extending it to open-ended quality is unsolved. See ../06_transformers_llm/07_reasoning_models.md.

Comparison

The 2026 pipeline at leading labs: large-scale SFT, then preference alignment via DPO-family or GRPO-family methods, then reasoning amplification with RLVR and test-time compute.

What can go wrong

• Reward hacking — the policy exploits the reward model rather than improving. Mitigate with the KL penalty, verifiable rewards where possible, and by monitoring the reward-model score against human judgement.
• Alignment tax — capability on some tasks drops after alignment. Mix in pretraining or SFT data during the RL stage.
• Sycophancy — annotators prefer agreeable answers, so the model learns to agree. A known artefact of preference data, and one reason calibration often worsens after RLHF. See ../04_model_evaluation/05_calibration.md.
• Mode collapse — reduced diversity as the policy concentrates on high-reward responses.

Do you need any of this?

For most application work, no. You are consuming aligned models, not producing them. Where it becomes relevant:

• DPO on a few thousand preference pairs to tune tone or format for a specific product — this is genuinely accessible with LoRA.
• RLVR if you have a verifiable domain and want a specialised model.
• Understanding the failure modes, because sycophancy, refusal behaviour and miscalibration all originate here and show up in your product.

Interview angle

• “Explain RLHF.” — SFT, then a reward model fit to human preference pairs, then policy optimisation against that reward with a KL penalty to the reference model. The KL term is what stops reward hacking and degenerate output.
• “What problem does DPO solve?” — it removes the reward model and the RL loop by exploiting the closed form of the KL-constrained optimum, so you train on preference pairs with a supervised-style loss and two models instead of four. The cost is that it’s offline and can’t explore.
• “What is GRPO and why did it displace PPO for reasoning?” — it drops the value critic by using the mean reward of a sampled group of responses to the same prompt as the baseline. That halves the memory and compute while staying on-policy, which is what long-chain reasoning training needs.
• “What is RLVR?” — RL where the reward is an automatic verifier rather than a learned model. No labelling bottleneck and no proxy to hack, but it only applies where correctness is cheaply checkable.
• “Why is the KL penalty necessary?” — without it the policy drifts arbitrarily far to maximise a proxy reward, producing text that scores well and reads badly. It anchors the policy near a model known to produce coherent language.
• “Why do aligned models sometimes become sycophantic and poorly calibrated?” — preference data rewards agreeable, confident-sounding answers, so the model learns that. It’s a direct consequence of optimising human preference rather than correctness.