Cross-Entropy
The cost of being wrong about a distribution — the loss function training most classifiers and every language model, and information theory's most-used export to ML.
Reviewed July 16, 2026Stable
When not to use it
- For regression with continuous targets. Cross-entropy is for distributions over classes or tokens; squared error and its kin fit continuous outputs.
- When you need calibrated probabilities out of the box. Low cross-entropy doesn't guarantee calibration; that needs separate treatment.
- With extreme class imbalance, unmodified. The dominant class can swamp the loss; reweighting or focal loss is often needed.
Reach for something else instead
- Focal loss down-weights easy examples for imbalanced detection and classification.
- Hinge loss (as in SVMs) optimises a margin rather than a probability.
- Mean squared error is the counterpart for continuous targets.
Sources & further reading
- Shannon (1948), A Mathematical Theory of Communication — the information-theoretic foundation cross-entropy rests on.
- Goodfellow, Bengio & Courville, Deep Learning — derives cross-entropy loss and its equivalence to maximum likelihood.
- Szegedy et al. (2016), Rethinking the Inception Architecture — introduced label smoothing as a cross-entropy regulariser.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating cross-entropy and KL divergence as interchangeable. Cross-entropy includes the data's own entropy; KL is only the excess.
- Expecting low loss to mean good calibration. It doesn't; a model can be confidently, cheaply wrong in aggregate.
- Ignoring a few catastrophic errors because accuracy looks fine. Cross-entropy is dominated by confident mistakes accuracy hides.
At a glance
FieldFoundations
FormulaH(p,q) = −Σ p log q
Equalsentropy + KL divergence
Same asmaximum-likelihood training
DifficultyIntermediate