Policy Gradient
Learning the behaviour directly instead of learning values — the method behind RLHF, and its entire difficulty is variance.
When not to use it
- Without a baseline. REINFORCE alone has variance so high it barely learns.
- When samples are precious. On-policy means every update discards your data.
- With large steps. A collapsed policy generates the data for the next update. There's no reloading.
- When Q-learning fits. Discrete actions, cheap samples, off-policy data — Q-learning is more sample-efficient.
Reach for something else instead
- Q-learning — off-policy, more sample-efficient, discrete actions only.
- PPO — policy gradient with a trust region. The practical default.
- DPO — for preference learning, removes the RL entirely.
- Evolution strategies — no gradients at all, embarrassingly parallel, sample-hungry.
Sources & further reading
- Williams (1992), Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning — REINFORCE; the base algorithm.
- Sutton et al. (2000), Policy Gradient Methods for Reinforcement Learning with Function Approximation — the theorem; no environment gradient needed.
- Schulman et al. (2016), High-Dimensional Continuous Control Using Generalized Advantage Estimation — GAE; the bias-variance knob everyone uses.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Using raw returns instead of advantages. Subtracting a baseline is free variance reduction with no bias.
- Reusing off-policy data without importance correction. The theorem's expectation is under the current policy.
- Taking large policy steps. A collapse is unrecoverable because the broken policy produces your next batch.
- Reaching for policy gradients on a discrete, cheap, off-policy problem where Q-learning wins.
At a glance
FieldFoundations
What it learnsthe behaviour directly, not values
The whole problemvariance; the credit assignment is a Monte Carlo guess
The free fixsubtract a baseline; unbiased, big variance cut
Why LLMs use itthe action space is the vocabulary, and the model is already a policy
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Policy gradient vs. Q-learning — one learns behaviour directly and handles any action space at high variance; the other learns values, is more sample-efficient, and can't do continuous actions.