Interpretability
Working out what's actually happening inside a model — distinct from explainability, much harder, and the only approach that could tell you what a system will do before it does it.
When not to use it
- As a substitute for an interpretable model. For high-stakes tabular decisions, Rudin's argument holds: use something readable.
- As a deployment gate. It's research. It won't tell you your model is safe this quarter.
- Probing as evidence of use. Finding information in activations doesn't mean the model uses it.
- Reading neurons directly. Superposition means they respond to unrelated things by design.
Reach for something else instead
- Inherently interpretable models — decision trees, linear models, GAMs. For high-stakes applied work, often the right answer.
- Explainability methods — cheaper, post-hoc, and they explain the explanation.
- Behavioural evaluation — tests what it did on inputs you thought of. Necessary, insufficient.
- Steering vectors — the practical spillover; useful now.
Sources & further reading
- Elhage et al. (2022), Toy Models of Superposition — why neurons aren't the unit; features outnumber dimensions.
- Templeton et al. (2024), Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — sparse autoencoders at production scale; features that are legible and steerable.
- Rudin (2019), Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead — the argument against the whole approach, for the cases where it applies.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Conflating it with explainability. One tells a story about a decision; the other finds the mechanism.
- Expecting neurons to be interpretable. Superposition means they aren't, on purpose.
- Treating probe accuracy as evidence the model relies on that information.
- Assuming feature-level understanding gives behaviour-level understanding. That step isn't established.
At a glance
FieldSafety & Ethics
Distinct from explainabilitymechanism, not story
The core obstaclesuperposition; features outnumber dimensions
Best current toolsparse autoencoders
The practical spilloversteering vectors
Honest statuslosing the race against scale
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Interpretability vs. explainability — one asks what the network computes, the other asks for a plausible account of one decision. Only the first can be checked.