Explainability
Getting a model to show its working — and the uncomfortable fact that most methods explain the explanation, not the decision.
Reviewed July 10, 2026Stable
When not to use it
- As a substitute for an interpretable model in high-stakes settings. If the decision affects someone's liberty, health, or livelihood, an approximation of a reason is not a reason.
- To satisfy a regulator without understanding the method's assumptions. An explanation you can't defend under questioning is worse than admitting the model is opaque.
- On correlated features, taking attributions at face value. SHAP's independence assumptions break, and the numbers still print.
Reach for something else instead
- Intrinsically interpretable models — decision trees, scorecards, generalised additive models. You lose some accuracy and gain an answer you can actually stand behind.
- Counterfactuals — "you'd have been approved with £3k more income" is more useful to a person than a ranked feature list.
- Rigorous testing by subgroup — sometimes what you need isn't why one decision happened, but evidence about how the system behaves across people.
Read more on the blog
- AI in finance: the regulator looked and stepped backBanking has had formal model risk regulation since 2011. In April 2026 the successor framework arrived and deliberately placed generative AI outside its scope. That decision is the finding.
- How to read an AI paperA meaningful fraction of state-of-the-art results, in the highest-prestige venues, could not be reproduced from the published artifact. The failures were structural and they are visible from the outside if you know where to look.
- Neurosymbolic AI: what the symbolic half gets rightOn a structured manipulation task in early 2026, a neurosymbolic system scored 95% against 34% for a fine-tuned vision-language model, using less energy. Symbolic AI lost the last argument on cost, not on merit.
- Does it understand? The argument, properly statedBoth sides of this debate are usually presented by their opponents. Here is the sceptical case at full strength, the case for at full strength, why the two keep missing each other, and what would actually settle it.
Further reading
- Rudin (2019), Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead — the strongest argument in the field, and the one most often ignored.
- Adebayo et al. (2018), Sanity Checks for Saliency Maps — several popular methods fail basic tests. Read this before trusting a heatmap.
- Turpin et al. (2023), Language Models Don't Always Say What They Think — stated reasoning can be plausible and unfaithful.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading feature importance as causation. It describes the model's behaviour, not the world's mechanics.
- Trusting a saliency map because it looks convincing. Convincing is what they're optimised for; several fail randomisation tests.
- Reporting one method's output as the explanation, when a different method would have named different features.
At a glance
FieldSafety & Ethics
Core ideareconstructing a reason
Post-hoc methodsapproximate, unverifiable
Strongest optioninterpretable by construction
DifficultyIntermediate
Often compared with
Explainability vs. interpretability — a story about the decision vs. a model whose workings are the decision.
Where this sits
6 concepts come first. Understanding it opens up 2 more.
3Levelsteps in
6Needs firstconcepts
2Opens up1% of 310
1Areastays here
Computed from the prerequisite graph, not assigned. How this works