DPO
Preference training without a reward model or reinforcement learning — the derivation that made RLHF simple, and it may not be free.
When not to use it
- With off-policy preference data. You're teaching the model about responses it wouldn't produce. Generate on-policy, then rank.
- Without watching the chosen response's likelihood. It can fall along with the rejected one. The loss won't tell you.
- On length-biased data, unregularised. Your model gets wordier and the metric improves.
- Assuming it matches PPO at frontier scale. The evidence is mixed and the labs' behaviour is a signal.
Reach for something else instead
- PPO / RLHF — online exploration, more complexity, possibly better at scale.
- KTO — binary feedback, no pairs. Matches what products actually collect.
- ORPO / SimPO — no reference model, fewer moving parts.
- Instruction tuning alone — if you don't have preference data, don't invent it.
Sources & further reading
- Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model — the derivation.
- Xu et al. (2024), Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study — the careful comparison; the gap may be real.
- Ethayarajh et al. (2024), KTO: Model Alignment as Prospect Theoretic Optimization — dropping the pairs requirement, which matters for real feedback.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Not monitoring the chosen response's likelihood. The gap can widen while both drop.
- Using preference data your model didn't generate. On-policy is the main lever.
- Ignoring length bias, then celebrating a wordier model.
- Treating the simplification as free. The variants exist because the original has pathologies.
At a glance
FieldLanguage & LLMs
What it removesthe reward model and the RL loop
The insightthe LM is secretly its own reward model
The knobβ, how far it may drift from the reference
The pathologyboth likelihoods can fall while the gap widens
Unsettledwhether it matches PPO at scale
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
DPO vs. PPO — one is a supervised loss on pairs you already have; the other explores online with a reward model. The first is far simpler and possibly slightly worse.