Mutual Information
How much knowing one thing tells you about another — the general measure of dependence that captures relationships correlation misses.
Reviewed July 16, 2026Stable
When not to use it
- On continuous high-dimensional data without care. Estimates are badly biased; a confident MI number there is often an artefact.
- When a linear relationship is all you expect. Correlation is cheaper, better-understood, and sufficient.
- As proof of causation. Mutual information is symmetric and says nothing about direction or cause — only shared information.
Reach for something else instead
- Correlation is simpler and adequate when the relationship is linear.
- Distance correlation captures nonlinear dependence with better-behaved estimation than MI in some settings.
- Conditional independence tests are the right tool when the question is really about causal structure.
Sources & further reading
- Shannon (1948), A Mathematical Theory of Communication — defined mutual information alongside entropy.
- Belghazi et al. (2018), Mutual Information Neural Estimation (MINE) — neural estimators for MI, and the wave of methods built on them.
- Tschannen et al. (2020), On Mutual Information Maximization for Representation Learning — showed the MI-maximisation justification for contrastive learning is looser than claimed.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Trusting MI estimates on high-dimensional continuous data. The bias inflates the number and manufactures dependence.
- Reading mutual information as causation. It's symmetric; it cannot tell you which variable drives which.
- Assuming zero correlation means independence. It doesn't — MI can be large where correlation is zero.
At a glance
FieldFoundations
FormulaKL between joint and product of marginals
Symmetric?Yes
Zero whenvariables are independent
Catchesany dependence, not just linear
DifficultyAdvanced