Actor-Critic
The reinforcement-learning architecture that pairs a decision-maker with a judge — one network chooses actions, another scores them — and underpins most modern deep RL.
Reviewed July 16, 2026Stable
When not to use it
- On small, discrete problems where a simple value method (Q-learning) suffices — actor-critic's machinery is overkill.
- When training stability is paramount and you can't afford to tune two coupled networks — simpler or preference-based methods may be safer.
- When a non-RL approach fits the problem — actor-critic is for sequential decision-making under reward, not supervised tasks.
Reach for something else instead
- Q-learning / value methods for discrete, smaller action spaces.
- Pure policy gradients (REINFORCE) when simplicity matters more than variance.
- DPO and direct preference methods for RLHF-style alignment without the RL loop.
Read more on the blog
- What is reinforcement learning? Learning from rewardReinforcement learning went from a niche corner of AI obsessed with games and robots to the paradigm that shapes how every modern language model behaves. Here's what it actually is, learning by trial, reward, and consequence, why it's different from other machine learning, and how it quietly became the layer between a smart model and a useful one.
- How AI models are trained: from raw text to a system that helpsA language model isn't programmed, it's grown, in stages, from a firehose of text into a system that answers helpfully. The full modern pipeline: pretraining, supervised fine-tuning, preference alignment, and the reasoning training that defines 2026, what each stage does, why none can be skipped, and how the recipe changed.
Sources & further reading
- Sutton & Barto, Reinforcement Learning: An Introduction — the canonical treatment of actor-critic methods.
- Mnih et al. (2016), Asynchronous Methods for Deep Reinforcement Learning — A3C, actor-critic at scale.
- Schulman et al. (2017), Proximal Policy Optimization Algorithms — PPO, the actor-critic method behind RLHF.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Letting actor and critic get out of sync — an inaccurate critic gives the actor bad advantage signals.
- Reaching for actor-critic on problems a simple value method would solve more reliably.
- Underestimating PPO's tuning sensitivity, then blaming the method when it diverges.
At a glance
FieldFoundations (RL)
Actorchooses actions
Criticevaluates them (advantage)
Key benefitlow-variance policy learning
Famous usePPO / RLHF
DifficultyAdvanced