KL Divergence
A measure of how far one distribution is from another — not a distance, but the workhorse behind variational inference, RLHF, distillation, and diffusion.
Reviewed July 16, 2026Stable
When not to use it
- As a distance. It's asymmetric and violates the triangle inequality; treating it as a metric produces wrong reasoning.
- When supports don't overlap. If q is zero where p isn't, KL is infinite — Wasserstein or a smoothed variant is safer.
- For symmetric "how different are these two" questions. Use a symmetric measure (Jensen–Shannon, Wasserstein) when neither distribution is privileged.
Reach for something else instead
- Jensen–Shannon divergence symmetrises KL and stays finite.
- Wasserstein distance gives a true metric with a meaningful geometry, better-behaved for generative models.
- Total variation distance bounds how differently two distributions can weight any event.
Sources & further reading
- Kullback & Leibler (1951), On Information and Sufficiency — the original definition.
- Blei, Kucukelbir & McAuliffe (2017), Variational Inference: A Review for Statisticians — how reverse-KL minimisation underlies modern approximate inference.
- Hinton, Vinyals & Dean (2015), Distilling the Knowledge in a Neural Network — distillation as matching distributions.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Calling it a distance. The asymmetry is the whole point and ignoring it leads to real errors.
- Forgetting the infinity. Zero probability under q where p is positive makes KL blow up — a common training instability.
- Mixing up the directions. Forward KL is mean-seeking (blurry), reverse KL is mode-seeking (narrow); they produce different models.
At a glance
FieldFoundations
FormulaΣ p log(p/q)
Symmetric?No
Zero whenp = q
PowersVAEs, RLHF, distillation, diffusion
DifficultyAdvanced