Interpretability
Working out what's actually happening inside a model — distinct from explainability, much harder, and the only approach that could tell you what a system will do before it does it.
Reviewed July 13, 2026Stable
When not to use it
- As a substitute for an interpretable model. For high-stakes tabular decisions, Rudin's argument holds: use something readable.
- As a deployment gate. It's research. It won't tell you your model is safe this quarter.
- Probing as evidence of use. Finding information in activations doesn't mean the model uses it.
- Reading neurons directly. Superposition means they respond to unrelated things by design.
Reach for something else instead
- Inherently interpretable models — decision trees, linear models, GAMs. For high-stakes applied work, often the right answer.
- Explainability methods — cheaper, post-hoc, and they explain the explanation.
- Behavioural evaluation — tests what it did on inputs you thought of. Necessary, insufficient.
- Steering vectors — the practical spillover; useful now.
Read more on the blog
- Mechanistic interpretability: opening the AI black boxWe built AI systems that work without fully understanding how they work. We have every number inside them, yet the numbers do not obviously mean anything. Mechanistic interpretability is the effort to reverse-engineer that black box, and it has started to succeed, revealing why you cannot just read a neuron and how researchers now extract readable concepts from the tangle.
- AI alignment and safety, without the hype or dismissalAI safety is discussed either as impending doom or as overblown hype, and neither framing helps you understand it. The actual landscape: what alignment means, the concrete risks experts agree on, the speculative ones they don't, and why serious people land in very different places.
- AI vs machine learning vs deep learning: the differenceThese three terms are used interchangeably and are not interchangeable. They are nested: deep learning sits inside machine learning, which sits inside artificial intelligence. Knowing which circle you are in tells you what to expect about data, cost, transparency, and how the system will fail.
- What is the Turing test? And why it stopped matteringTuring never proposed the imitation game as a definition of thinking. He proposed it to replace a question he considered meaningless. Machines have now passed versions of it, sometimes judged more human than actual humans, and the striking thing is how little that settled.
Further reading
- Elhage et al. (2022), Toy Models of Superposition — why neurons aren't the unit; features outnumber dimensions.
- Templeton et al. (2024), Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet — sparse autoencoders at production scale; features that are legible and steerable.
- Rudin (2019), Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead — the argument against the whole approach, for the cases where it applies.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Conflating it with explainability. One tells a story about a decision; the other finds the mechanism.
- Expecting neurons to be interpretable. Superposition means they aren't, on purpose.
- Treating probe accuracy as evidence the model relies on that information.
- Assuming feature-level understanding gives behaviour-level understanding. That step isn't established.
At a glance
FieldSafety & Ethics
Distinct from explainabilitymechanism, not story
The core obstaclesuperposition; features outnumber dimensions
Best current toolsparse autoencoders
The practical spilloversteering vectors
Honest statuslosing the race against scale
DifficultyAdvanced
Often compared with
Interpretability vs. explainability — one asks what the network computes, the other asks for a plausible account of one decision. Only the first can be checked.
Where this sits
10 concepts come first. Understanding it opens up 1 more.
5Levelsteps in
10Needs firstconcepts
1Opens up0% of 310
1Areastays here
Computed from the prerequisite graph, not assigned. How this works