PPO
The policy gradient method that trains language models — and a careful study found its gains came from the implementation details, not the idea in the paper.
When not to use it
- Reimplemented from the paper. The code-level details are the performance and they aren't in it.
- When DPO fits. For preference learning, DPO removes the RL entirely and gets most of the way.
- When samples are expensive. On-policy means resampling constantly.
- Without a KL leash, in RLHF. The reward model gets hacked under pressure. That's what the penalty is for.
Reach for something else instead
- DPO — no RL, no reward model. Simpler; possibly slightly worse at scale.
- TRPO — the principled version; with the same code-level tricks, comparable.
- RL on verifiable rewards — PPO-family with a real reward. Where the frontier is.
- A reference implementation — genuinely: use one rather than writing your own.
Sources & further reading
- Schulman et al. (2017), Proximal Policy Optimization Algorithms — the paper; the clipped objective.
- Engstrom et al. (2020), Implementation Matters in Deep RL: A Case Study on PPO and TRPO — the gains came from code-level optimisations, not the clipping.
- Henderson et al. (2018), Deep Reinforcement Learning that Matters — seeds, implementations and hyperparameters dominate. The context for the above.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Citing the clipped objective as the reason it works. Engstrom et al. showed it isn't.
- Reimplementing from the paper and wondering why it underperforms. The tricks aren't published.
- Confusing PPO's clipping with RLHF's KL penalty. Different mechanisms; the second is a leash against reward hacking.
- Treating deep RL results as reproducible across implementations. Henderson et al. say otherwise.
At a glance
FieldFoundations
What it ispolicy gradient with a clipped trust region
What trains LLMsthis, via RLHF
The findingits gains came from unpublished code-level optimisations, not the clipping (Engstrom et al.)
Why the KL penalty existsthe reward model gets hacked under pressure
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
PPO vs. TRPO — one is a few lines and one is constrained optimisation, and with the same code-level tricks they perform comparably. The simplicity won; the credit went to the wrong place.