Q-Learning
Learning the value of every action in every state, by bootstrapping off your own estimates — which converges beautifully in theory and diverges in practice.
When not to use it
- With continuous actions. You'd have to maximise over a continuum every step. Use policy gradients.
- When samples are expensive. It needs tens of millions. That's why RL lives in simulation.
- Without a target network and replay. The deadly triad will diverge, not just underperform.
- Expecting the tabular guarantees. A neural network voids them entirely.
Reach for something else instead
- Policy gradient / PPO — continuous actions, and what LLM work uses.
- Model-based RL — far better sample efficiency, more machinery.
- Contextual bandits — if actions don't affect future states, this is much simpler.
- Imitation learning — if you have demonstrations, copying is cheaper than exploring.
Sources & further reading
- Watkins & Dayan (1992), Q-learning — the convergence proof; bootstrapping off your own estimates works.
- Mnih et al. (2015), Human-level control through deep reinforcement learning — DQN; Atari from pixels, and the two tricks that made it stable.
- van Hasselt, Guez & Silver (2016), Deep Reinforcement Learning with Double Q-learning — the max operator is biased upward; decouple selection from evaluation.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Expecting convergence with a neural network. The proof is for tables; the triad is right there.
- Skipping the target network. You're regressing toward a target that moves when you update.
- Ignoring overestimation. The max over noisy estimates is biased upward, systematically.
- Reaching for RL when a bandit fits. If your actions don't change the next state, this is enormous overkill.
At a glance
FieldFoundations
What it learnsthe long-run value of every action in every state
The trickbootstrap: update a guess using a guess
Why it's off-policythe max means you learn the greedy policy while behaving however
The deadly triadapproximation + bootstrapping + off-policy can diverge
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Tabular vs. deep Q-learning — one has a convergence proof, the other has the deadly triad and a bag of tricks. Only the second scales.