Explainability
Getting a model to show its working — and the uncomfortable fact that most methods explain the explanation, not the decision.
When not to use it
- As a substitute for an interpretable model in high-stakes settings. If the decision affects someone's liberty, health, or livelihood, an approximation of a reason is not a reason.
- To satisfy a regulator without understanding the method's assumptions. An explanation you can't defend under questioning is worse than admitting the model is opaque.
- On correlated features, taking attributions at face value. SHAP's independence assumptions break, and the numbers still print.
Reach for something else instead
- Intrinsically interpretable models — decision trees, scorecards, generalised additive models. You lose some accuracy and gain an answer you can actually stand behind.
- Counterfactuals — "you'd have been approved with £3k more income" is more useful to a person than a ranked feature list.
- Rigorous testing by subgroup — sometimes what you need isn't why one decision happened, but evidence about how the system behaves across people.
Sources & further reading
- Rudin (2019), Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead — the strongest argument in the field, and the one most often ignored.
- Adebayo et al. (2018), Sanity Checks for Saliency Maps — several popular methods fail basic tests. Read this before trusting a heatmap.
- Turpin et al. (2023), Language Models Don't Always Say What They Think — stated reasoning can be plausible and unfaithful.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reading feature importance as causation. It describes the model's behaviour, not the world's mechanics.
- Trusting a saliency map because it looks convincing. Convincing is what they're optimised for; several fail randomisation tests.
- Reporting one method's output as the explanation, when a different method would have named different features.
At a glance
FieldSafety & Ethics
Core ideareconstructing a reason
Post-hoc methodsapproximate, unverifiable
Strongest optioninterpretable by construction
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Explainability vs. interpretability — a story about the decision vs. a model whose workings are the decision.