Mechanistic interpretability: opening the AI black box

Modern AI has an unusual property for an engineered technology: we built it, it works, and we do not fully understand how. This is not because the internals are hidden. We have every single number inside a neural network, all the weights and all the activations, available to inspect. The problem is that the numbers do not obviously mean anything. A model is billions or trillions of numbers that collectively produce fluent language and competent reasoning, with no labels explaining which number does what. We can watch the machine think in complete detail and still have no idea what it is thinking.

Mechanistic interpretability is the research field trying to fix that: to reverse-engineer the internal computations of a neural network, to work out not just that it produces the right output but how, in terms a human can understand. It is one of the most important efforts in AI, because a system we cannot inspect is a system we cannot fully trust, audit, or make safe. This piece explains what the black box problem actually is, why you cannot understand a network by reading its neurons one at a time, the surprising reason its internal concepts are tangled together, the tool that has started to untangle them, and the honest state of how much we can and cannot yet see. It has moved, in the last few years, from philosophy to a working science.

The black box is not hidden, it is unlabelled

It helps to be precise about what the black box problem is and is not. It is not that the model's internals are secret or inaccessible. For an open model you can read every parameter; even for a closed one, the people who built it can. The difficulty is that having the numbers is not the same as understanding them. Knowing that a particular value deep in the network is 0.73 tells you nothing about what role it plays in the model deciding to answer a question one way rather than another.

The knowledge a model has learned is not stored in a readable form. It is distributed across the interactions of billions of numbers, encoded in a way that emerged from training rather than being designed to be legible. So the black box is really an unlabelled box: everything is visible, nothing is annotated, and the task of interpretability is to discover the labels, to map regions of this vast numerical object onto concepts and computations a person can follow. That mapping is what would let us say, of a given output, here is why the model produced it.

Why you cannot just read the neurons

The natural first hope is that the network is organised the way you might design it: that each neuron corresponds to one concept, so you could read the network by checking which neuron lights up for what. Find the "dog" neuron, the "France" neuron, the "sarcasm" neuron, and you have decoded the model. Early work found a few neurons that did seem to track single concepts, which was encouraging.

But it mostly does not work, and the reason has a name: polysemanticity. Most individual neurons are not dedicated to one concept; they fire for several, often completely unrelated ones. A single neuron might activate for the idea of legal precedent, and also for academic citations, and also for recipe ingredients, three things with nothing obvious in common. Reading such a neuron tells you almost nothing, because its activation could mean any of several unrelated things depending on context. When most of the network is built from polysemantic neurons like this, examining it one neuron at a time is hopeless. The unit you can see, the neuron, is not the unit of meaning. This is the obstacle that stalled interpretability for years, and understanding why it happens is the key that eventually got past it.

Superposition: why the concepts are tangled

The reason neurons are polysemantic turns out to be one of the more surprising findings in the field, and it is called superposition. A model needs to represent far more distinct concepts, or features, than it has neurons to devote to them. But a space with a fixed number of dimensions can only hold that many truly independent directions: a layer with a thousand neurons has only a thousand orthogonal directions to work with. If the model wants to represent tens of thousands of features, it cannot give each its own clean direction.

So it cheats, cleverly. It packs many features into the same space as overlapping, non-perpendicular directions, storing far more concepts than dimensions by letting them share and interfere. This works because features are sparse, meaning only a few are active at any one time, so the overlaps rarely collide in practice. The model gets to compress a huge vocabulary of concepts into a limited number of neurons, which is efficient for it, and this compression is exactly what produces polysemanticity: because many features are superimposed onto the same neurons, any single neuron participates in representing several of them, and so appears to fire for unrelated concepts. The tangle we see is the shadow of the model squeezing more meaning into its neurons than a one-concept-per-neuron scheme would allow. Superposition is why the box looks scrambled, and recognising it reframed the problem: the goal is not to read neurons but to undo the superposition and recover the underlying features.

Sparse autoencoders: untangling the features

The tool that has driven the field's recent progress does exactly that. A sparse autoencoder is a second, separate neural network trained to take the model's tangled internal activations and decompose them into a much larger set of cleaner features, each of which activates only rarely. The idea, borrowed from a classical technique called dictionary learning, is to find a big dictionary of underlying features such that any activation in the model can be reconstructed as a combination of just a few of them. Because the dictionary is large and the combinations are sparse, the features it finds tend to be monosemantic: each one corresponds to a single, human-interpretable concept, in a way the raw neurons did not.

The results have been striking. Applied to real language models, sparse autoencoders recover features corresponding to recognisable concepts, from concrete things like specific landmarks or programming constructs to abstract ones like a text being about inner conflict or a piece of code containing a security vulnerability. Make the dictionary bigger and you find progressively finer, more specific features. For the first time, this turns a slab of meaningless activations into something like a readable list of the concepts the model is using at a given moment. It is decompression: taking the superimposed features the model packed together and spreading them back out into separable, nameable pieces.

Features you can steer, and circuits you can trace

Two further developments turned this from an observation into evidence and a potential tool. The first is that these features are not just correlations; they are causal. Once you have identified the feature representing a concept, you can intervene on it, artificially amplifying or suppressing it, and the model's behaviour changes in the corresponding way. Turn up a feature for a particular landmark and the model starts steering everything it says toward that landmark; turn up a feature for a certain tone and its outputs take on that tone. This steering both proves the features are real, since manipulating them has predictable effects, and hints at a way to control model behaviour directly at the level of concepts rather than through prompting.

The second is the move from individual features to circuits: connected chains of computation that implement a specific behaviour. Interpretability researchers have identified, for example, mechanisms that carry out in-context pattern completion, letting a model continue a sequence it has seen before, and have traced small end-to-end computations that a model uses to answer particular kinds of question. Tools that build attribution graphs can now trace a path from a model's input, through the internal features it activates, to its output, offering something like a wiring diagram of a specific piece of reasoning. This is the shift from asking what does the model represent to how does it compute, which is the deeper goal the field's name points at.

Where things stand, honestly

It is worth being clear-eyed about how far this has and has not come, because the topic attracts both hype and dismissal. On the progress side, mechanistic interpretability has moved from a largely theoretical pursuit to a working practice. Sparse autoencoders have been scaled up to frontier-size models. Interpretability tools have been used in real pre-deployment safety evaluations of deployed models, to check for concerning internal features before release. The work spans many groups across industry and universities, and the field was recently named a breakthrough technology by MIT, a sign of how seriously it is now taken.

But the honest picture is one of partial vision, not a solved box. We can now read some of what a model is doing, not all of it. Serious challenges remain unresolved. Scaling these methods to fully cover the largest models is hard and incomplete. Validating that a discovered feature or circuit is truly what it appears to be is very difficult, and there is a real cautionary finding that sparse autoencoders can produce plausible-looking, interpretable-seeming features even when run on randomly initialised networks that compute nothing meaningful, which means an interpretable-looking result is not automatically a true one. Features can split or blur in ways that complicate clean interpretation. And the ultimate test, showing that interpretability reliably makes models safer rather than just more legible, is still being worked out. The flashlight is real and improving, but it lights a portion of the cave, not the whole of it.

Why it matters

The reason to care is that understanding is the foundation of trust and safety. A model we can only evaluate by its outputs can hide things: it could harbour a capability it does not display, a bias that only surfaces in rare cases, or, in the worst case discussed in alignment research, an intention to behave differently when it thinks it is not being watched. Testing behaviour from the outside can never fully rule these out, because a system can produce the right outputs for the wrong internal reasons. Interpretability offers the possibility of checking the reasons directly: of auditing a model's internals for deception or dangerous capability, of catching misalignment before deployment rather than after an incident, and of grounding our trust in a model in something more than the fact that it has behaved well so far. If AI systems are going to be given real responsibility, being able to see inside them, rather than only watching what they do, may be what makes that safe.

The short version

Mechanistic interpretability tries to reverse-engineer how a neural network computes its outputs, turning the model's billions of unlabelled numbers into human-understandable concepts and computations. The black box is not hidden but unlabelled: we can see every weight, but the knowledge is distributed and not stored legibly. You cannot understand the network by reading neurons one at a time, because most neurons are polysemantic, firing for several unrelated concepts. This happens because of superposition: the model packs more features than it has neurons into overlapping directions, exploiting the fact that features are rarely active at once. Sparse autoencoders untangle this by decomposing activations into a large, sparse set of cleaner, monosemantic features, which can be causally steered, and researchers are now tracing circuits that implement specific behaviours. The field has become a working science but still sees only part of the picture.

The idea to hold onto is that a neural network is not a hidden box but an unlabelled one, whose concepts are compressed together through superposition, and interpretability is the science of decompressing them into features and circuits we can read, so that we can eventually understand, audit, and trust these systems from the inside rather than only judging them by their behaviour. We built minds we cannot yet read, and learning to read them may be as important as building them.

Common questions

What is mechanistic interpretability? Mechanistic interpretability is a research field that tries to reverse-engineer the internal workings of neural networks, to understand not just what output a model produces but how it computes that output, in terms a human can follow. The challenge is that a model's knowledge is stored as billions of unlabelled numbers whose meaning is not obvious. Interpretability aims to map those numbers onto understandable concepts and computations, so that a model's reasoning can be inspected directly. It matters because a system we cannot understand is one we cannot fully audit, trust, or make safe.

Why can't we understand AI by reading its neurons? Because most neurons are polysemantic: a single neuron fires for several unrelated concepts, such as legal precedent, academic citations, and recipe ingredients all at once. Its activation could mean any of them depending on context, so reading it in isolation reveals almost nothing. Early work found a few neurons that tracked single concepts, but the majority do not, so examining the network one neuron at a time fails. The neuron, the unit we can easily see, is not the unit of meaning, which is the core obstacle interpretability had to get past.

What is superposition in neural networks? Superposition is the phenomenon where a model represents more distinct concepts, called features, than it has neurons, by packing them into overlapping, non-perpendicular directions in its activation space rather than giving each its own clean direction. It works because features are sparse, only a few active at once, so the overlaps rarely collide in practice. Superposition lets the model compress far more meaning into a limited number of neurons, which is efficient, but it is the direct cause of polysemanticity: because many features share the same neurons, each neuron appears to fire for several unrelated concepts.

What is a sparse autoencoder in interpretability? A sparse autoencoder is a separate neural network trained to decompose a model's tangled internal activations into a much larger set of cleaner features, each of which activates only rarely. Based on the idea of dictionary learning, it finds a large dictionary of underlying features such that any activation can be reconstructed from just a few of them. Because the dictionary is large and the combinations sparse, the recovered features tend to be monosemantic, each corresponding to a single interpretable concept, unlike the raw polysemantic neurons. Sparse autoencoders are the main tool that has driven recent progress in reading models.

Can you control an AI by changing its internal features? To a degree, yes, and this is one of the more striking findings. Once interpretability identifies the feature representing a concept, researchers can intervene on it, amplifying or suppressing that feature, and the model's behaviour changes accordingly. Amplifying a feature for a particular topic makes the model steer its outputs toward that topic, for example. This steering serves two purposes: it proves the features are causally real rather than coincidental, since manipulating them has predictable effects, and it suggests a way to control model behaviour directly at the level of concepts, complementing prompting and training.

How much of an AI model can we currently understand? Only part of it. Mechanistic interpretability has become a working science, with sparse autoencoders scaled to frontier models and interpretability tools used in real pre-deployment safety checks, and it was recently named a breakthrough technology by MIT. But we can read some of what a model does, not all. Major challenges remain: fully scaling the methods to the largest models, rigorously validating that discovered features and circuits are genuine, dealing with a finding that sparse autoencoders can produce interpretable-looking features even from meaningless random networks, and proving that interpretability actually improves safety. It is a partial and improving view, not a solved problem.

Why does mechanistic interpretability matter for AI safety? Because judging a model only by its outputs can never fully rule out hidden problems: a system can produce the right answers for the wrong internal reasons, harbour a capability it does not display, or, in the worst case, behave differently when it believes it is unobserved. Interpretability offers a way to check a model's internal reasoning directly, auditing for deception or dangerous capabilities and catching misalignment before deployment rather than after harm. It grounds trust in a model in something firmer than its past good behaviour. As AI systems take on more responsibility, being able to see inside them may be essential to doing so safely.