DPO
Preference training without a reward model or reinforcement learning — the derivation that made RLHF simple, and it may not be free.
Reviewed July 13, 2026Stable
When not to use it
- With off-policy preference data. You're teaching the model about responses it wouldn't produce. Generate on-policy, then rank.
- Without watching the chosen response's likelihood. It can fall along with the rejected one. The loss won't tell you.
- On length-biased data, unregularised. Your model gets wordier and the metric improves.
- Assuming it matches PPO at frontier scale. The evidence is mixed and the labs' behaviour is a signal.
Reach for something else instead
- PPO / RLHF — online exploration, more complexity, possibly better at scale.
- KTO — binary feedback, no pairs. Matches what products actually collect.
- ORPO / SimPO — no reference model, fewer moving parts.
- Instruction tuning alone — if you don't have preference data, don't invent it.
Read more on the blog
- How AI models are trained: from raw text to a system that helpsA language model isn't programmed, it's grown, in stages, from a firehose of text into a system that answers helpfully. The full modern pipeline: pretraining, supervised fine-tuning, preference alignment, and the reasoning training that defines 2026, what each stage does, why none can be skipped, and how the recipe changed.
- What is reinforcement learning? Learning from rewardReinforcement learning went from a niche corner of AI obsessed with games and robots to the paradigm that shapes how every modern language model behaves. Here's what it actually is, learning by trial, reward, and consequence, why it's different from other machine learning, and how it quietly became the layer between a smart model and a useful one.
Further reading
- Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model — the derivation.
- Xu et al. (2024), Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study — the careful comparison; the gap may be real.
- Ethayarajh et al. (2024), KTO: Model Alignment as Prospect Theoretic Optimization — dropping the pairs requirement, which matters for real feedback.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Not monitoring the chosen response's likelihood. The gap can widen while both drop.
- Using preference data your model didn't generate. On-policy is the main lever.
- Ignoring length bias, then celebrating a wordier model.
- Treating the simplification as free. The variants exist because the original has pathologies.
At a glance
FieldLanguage & LLMs
What it removesthe reward model and the RL loop
The insightthe LM is secretly its own reward model
The knobβ, how far it may drift from the reference
The pathologyboth likelihoods can fall while the gap widens
Unsettledwhether it matches PPO at scale
DifficultyAdvanced
Often compared with
DPO vs. PPO — one is a supervised loss on pairs you already have; the other explores online with a reward model. The first is far simpler and possibly slightly worse.
Where this sits
A destination. 28 concepts lead here, and nothing in the corpus depends on it.
14Levelsteps in
28Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstRLHF
Computed from the prerequisite graph, not assigned. How this works