RLHF (Reinforcement Learning from Human Feedback)
Training a model on human preferences rather than correct answers — the step that turned text predictors into assistants.
When not to use it
- When you have correct answers. If you can demonstrate the right output, supervised fine-tuning is simpler, cheaper, and more reliable. Preferences are for when correctness can't be written down.
- Without enough preference data. A reward model trained on a few hundred comparisons will confidently encode the noise.
- When labellers can't judge the task. If the model is better than the humans rating it, you're training toward their limits, not past them.
Reach for something else instead
- DPO — the same preference signal without the reward model or the RL machinery. The sensible default for most teams now.
- Supervised fine-tuning on good examples, when good examples exist.
- Prompting — a system prompt gets you a surprising amount of behaviour shaping for zero training.
Sources & further reading
- Christiano et al. (2017), Deep Reinforcement Learning from Human Preferences — the technique, before language models.
- Ouyang et al. (2022), Training language models to follow instructions with human feedback — InstructGPT, the paper that made assistants work.
- Rafailov et al. (2023), Direct Preference Optimization — preference training without the reward model or the RL loop.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating the reward model as ground truth. It's an approximation, and hard optimisation against it finds its flaws rather than human preference.
- Ignoring the KL penalty's role, then wondering why the policy drifted somewhere strange.
- Assuming preference-trained means aligned. It means it produces outputs that labellers preferred — including sycophancy, hedging, and length.
At a glance
FieldLanguage & LLMs
Core ideatrain on preferences, not answers
PipelineSFT, reward model, policy optimisation
Weak linkthe reward model
Modern shortcutDPO
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
RLHF vs. supervised fine-tuning — learning from which output was preferred vs. learning from the output that was correct.