Information Theory
Shannon's mathematics of surprise — a way to measure information in bits, which turns out to be the measure almost everything in machine learning is quietly optimising.
Reviewed July 16, 2026Stable
When not to use it
- As a theory of meaning. Shannon deliberately excluded semantics; information content and importance are different things, and conflating them causes confusion.
- Where the distributions are unknown and unestimable. Information-theoretic quantities need probabilities; on tiny or badly-estimated distributions the numbers are noise dressed as rigour.
- As the only lens. It explains what's being optimised, not always why the optimisation generalises — that needs other tools.
Reach for something else instead
- Statistical estimation theory answers "how good is my estimate" without the coding framing.
- Decision theory frames the same problems around costs and actions rather than bits.
- Bayesian probability covers much of the same ground with an emphasis on belief update rather than encoding.
Sources & further reading
- Shannon (1948), A Mathematical Theory of Communication — the founding paper; defined entropy, channel capacity, and the source coding theorem.
- Cover & Thomas, Elements of Information Theory — the standard graduate text tying the theory to statistics and learning.
- Tishby, Pereira & Bialek (2000), The Information Bottleneck Method — the framework later applied controversially to deep learning.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Confusing information with meaning. A random string has maximal information content and zero meaning; the theory measures the former.
- Reading entropy as disorder. It's a precise measure of uncertainty about outcomes, not a vague "messiness".
- Treating the numbers as robust on small samples. Estimated entropy and mutual information are biased and noisy when data is scarce.
At a glance
FieldFoundations
Founded byClaude Shannon, 1948
Unitthe bit
Core quantityentropy H(X) = −Σ p log p
DifficultyIntermediate