Bayesian Inference
Updating beliefs with evidence, according to the only rule that's coherent — mathematically settled, practically expensive, and the thing modern models are bad at.
Reviewed July 13, 2026Stable
When not to use it
- With lots of data. The likelihood swamps the prior; you get the same answer at more expense.
- When you can't justify the prior. You are inserting a belief and getting it back out — the frequentist objection is real.
- Exactly, at scale. The mathematics is settled; the arithmetic is intractable. Approximate.
- For causal questions, from observational data. Pearl's point: P(y|x) is not "x causes y", and no data volume bridges it.
Reach for something else instead
- Deep ensembles — crude, and they beat most sophisticated uncertainty methods.
- Conformal prediction — distribution-free coverage guarantees, no prior needed.
- Frequentist methods — assumptions hidden in the procedure instead of stated in a prior.
- Bayesian optimisation — the version everyone uses, for expensive black boxes.
Read more on the blog
- What an AI confidence score actually meansA model that says it is 90% sure can be right 60% of the time, and a perfectly calibrated one still tells you nothing about the answer in front of you. Two theories of probability, and why the gap matters.
- Why machine learning does not do error barsModels differing only by random seed showed 0.057% variance in accuracy and 28.9% in certified robustness. The variance is not uniform, and knowing where it concentrates matters more than demanding error bars everywhere.
- Regularization: L1, L2, dropout and what unites themL2 regularization is not analogous to a Gaussian prior. It is exactly a Gaussian prior, and the identity is provable. Six techniques taught separately turn out to be one idea in different clothes.
Further reading
- Pearl (1988), Probabilistic Reasoning in Intelligent Systems — Bayesian networks; what made probabilistic AI tractable.
- Pearl (2009), Causality — and why the first book wasn't enough. Correlation isn't causation, formally.
- Gelman et al. (2013), Bayesian Data Analysis — the standard practical reference.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Ignoring the base rate. At 0.1% prevalence, most positives from a 99% test are false. That's the prior, and it's most of the answer.
- Treating a model's
P(y|x)as calibrated uncertainty. It's a point estimate with the model-uncertainty thrown away. - Reaching for sophisticated Bayesian deep learning. Train five models and look at the spread; it usually wins.
- Reading correlation from observational data as causation. Pearl spent thirty years on this and it's still ignored.
At a glance
FieldMachine Learning
The ruleposterior ∝ likelihood × prior; the only coherent way to update beliefs
Since1763
Why it isn't everywhereexact computation is intractable
What made it usablePearl's Bayesian networks, exploiting conditional independence
The unanswered critiqueobservational data can't answer interventional questions
DifficultyIntermediate
Often compared with
Bayesian vs. frequentist — one states its prior where you can see it, the other hides its assumptions in the procedure. With enough data they agree, and the argument is about the case where they don't.
Where this sits
A starting point. Nothing needs to come before it.
0Levelstarting point
0Needs firstnothing
0Opens upnothing further
1Areastays here
Computed from the prerequisite graph, not assigned. How this works