Bayesian Inference
Updating beliefs with evidence, according to the only rule that's coherent — mathematically settled, practically expensive, and the thing modern models are bad at.
When not to use it
- With lots of data. The likelihood swamps the prior; you get the same answer at more expense.
- When you can't justify the prior. You are inserting a belief and getting it back out — the frequentist objection is real.
- Exactly, at scale. The mathematics is settled; the arithmetic is intractable. Approximate.
- For causal questions, from observational data. Pearl's point: P(y|x) is not "x causes y", and no data volume bridges it.
Reach for something else instead
- Deep ensembles — crude, and they beat most sophisticated uncertainty methods.
- Conformal prediction — distribution-free coverage guarantees, no prior needed.
- Frequentist methods — assumptions hidden in the procedure instead of stated in a prior.
- Bayesian optimisation — the version everyone uses, for expensive black boxes.
Sources & further reading
- Pearl (1988), Probabilistic Reasoning in Intelligent Systems — Bayesian networks; what made probabilistic AI tractable.
- Pearl (2009), Causality — and why the first book wasn't enough. Correlation isn't causation, formally.
- Gelman et al. (2013), Bayesian Data Analysis — the standard practical reference.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Ignoring the base rate. At 0.1% prevalence, most positives from a 99% test are false. That's the prior, and it's most of the answer.
- Treating a model's
P(y|x)as calibrated uncertainty. It's a point estimate with the model-uncertainty thrown away. - Reaching for sophisticated Bayesian deep learning. Train five models and look at the spread; it usually wins.
- Reading correlation from observational data as causation. Pearl spent thirty years on this and it's still ignored.
At a glance
FieldMachine Learning
The ruleposterior ∝ likelihood × prior; the only coherent way to update beliefs
Since1763
Why it isn't everywhereexact computation is intractable
What made it usablePearl's Bayesian networks, exploiting conditional independence
The unanswered critiqueobservational data can't answer interventional questions
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Bayesian vs. frequentist — one states its prior where you can see it, the other hides its assumptions in the procedure. With enough data they agree, and the argument is about the case where they don't.