Neural Network Interpretability

Mechanistic interpretability (mech interp) is the reverse-engineering of neural networks: identifying the internal units a model actually uses — neurons, attention heads, directions, subspaces, circuits — and showing causally how their interaction produces behaviour. What distinguishes it from black-box evaluation is the commitment to mechanisms that can be ablated, steered, patched and predicted, rather than to input–output correlations. ^{1 2}

The field pursues two goals at once: a scientific one (what a capability is, and how training produces it) and an assurance one (whether a model is deceptive, sycophantic or pursuing hidden goals in ways that output monitoring cannot reveal). ^{3 1} Its central technical obstacle has been polysemanticity — neurons that appear to respond to unrelated things — and its central theoretical dispute is what causes it. A 2026 essay by Nick Cammarata, a co-author of the field’s founding circuit papers, argues that the canonical evidence for the standard answer was misread, and that networks are far more legible than the field assumed. That claim is the organising controversy of this page. ^{2 4}

Two programmes

Programme	Starting point	Method	Success criterion	Characteristic failure
Enumerative	the hypothesis that the model has a fixed vocabulary of features	train a decomposition (sparse autoencoder, transcoder, block-sparse featurizer) and audit the learned dictionary	a complete list of features, with no important direction left unexplained	combinatorial explosion; dictionary units that are artefacts of the decomposition rather than mechanisms
Exploratory	a question about a behaviour	move across neurons, circuits and attribution graphs; conjecture, ablate, and test at several scales	an explanation that survives intervention and predicts new cases	residual unexplained neurons; no single global map

Cammarata argues the second is more resilient to whatever polysemanticity remains, and that the big feature list was always a means to an end rather than the goal. ² The enumerative programme supplied most of the field’s public methods between 2022 and 2025; the exploratory programme is what the 2026 essay claims has actually been working. ^{2 5}

Polysemanticity and superposition

A neuron is polysemantic when it responds to inputs that seem to have nothing in common. The most cited example is neuron 4e:55 of InceptionV1, which appeared to fire for cat faces and the fronts of cars. ^{2 6} Cammarata catalogues five possible causes of an apparent mixture, only one of which is superposition: ²

1. Bad tools — the visualisation or attribution method is misleading.
2. Bad explanations — a plausible story built on those tools.
3. Bad learning — the architecture, data or optimisation produced a genuinely poor solution.
4. Recursive generalisation — the neuron branched into related territory as it grew.
5. Superposition — unrelated things packed into one neuron because they rarely interfere.

The superposition hypothesis, in its most influential form, says that when a network runs out of neurons it stops grouping related things and starts folding unrelated things together, tolerating interference because the folded items rarely co-occur. The result would be a knot: many near-equally good packings, so which items share a neuron would carry almost no information, and the network would have to be “untangled” before it could be studied. ^{2 7}

The word covers several distinct claims, and much of the 2026 dispute turns on separating them: ^{2 8 9}

• More features than dimensions — a representation can encode more concepts than it has basis vectors. In Toy Models of Superposition this is the definition. ⁸
• Composition vs superposition — Olah’s informal note distinguishes reusing a representation across readers (composition, benign and generalising) from packing unrelated computations into the same capacity (superposition, the concerning case). ⁹
• Arbitrary packing of unrelated things — the mechanistic claim that a neuron learns unrelated jobs together. Cammarata argues this is the only version that would be genuinely bad news, and that it is the version with almost no real-network evidence. ²

The superposition era (2020–2024)

Features, circuits, universality (2020)

Zoom In set the research programme with three speculative claims: features are the fundamental unit of neural networks and correspond to directions; features are connected by weights into circuits; and analogous features and circuits form across models and tasks (universality). ⁶ It also supplied the two canonical images of the field — the cat/car neuron and a circuit in which a car feature appeared to be routed into a set of dog neurons — which were read as evidence that networks hide concepts in high-dimensional combinations. ^{6 2}

Toy models (2022)

Toy Models of Superposition trained small ReLU networks on sparse synthetic features and showed that a model will represent more features than it has dimensions, accepting interference that a nonlinearity can filter out. Sparsity is what makes the packing worthwhile; the paper describes real networks as “noisily simulating larger, highly sparse networks”. ⁸ This is the theoretical backbone of the superposition picture: the argument is about storage capacity and interference, not about what any particular neuron means.

Dictionary learning (2023–2024)

If neurons are arbitrarily packed, the remedy is to learn a better basis. Towards Monosemanticity trained a sparse autoencoder (SAE) on a one-layer transformer and extracted a large dictionary of features, reporting that the median neuron scored 0 out of 14 on their interpretability rubric — the raters could form no hypothesis at all. ⁷ Scaling Monosemanticity extended the method to Claude 3 Sonnet and reported highly abstract features that were multilingual, multimodal, and causally effective under steering, including safety-relevant features for deception, sycophancy, bias, dangerous content and vulnerable code — with the explicit caution that a feature’s existence is not evidence that the behaviour occurs. ⁵

The strongest empirical case for the enumerative programme came from Liv Gorton’s application of SAEs to InceptionV1’s early vision: SAEs recovered additional curve detectors that were “missing” from the neuron basis, and decomposed some polysemantic neurons into more monosemantic constituents. ¹⁰ That result is directly contested by the 2026 essay (see below).

Composition vs superposition (2023)

Olah’s note reframed the issue as a trade-off between two ways of being distributed. Composition lets many readers reuse the same representation and generalises well; superposition packs unrelated computations and does not. The note calls a mere rotation away from the neuron basis a “trivial” form of distribution that “doesn’t change the fundamental geometry”. ⁹ Cammarata calls this framing the closest to his own, while noting that most researchers and agents still use “superposition” to mean the feature-count claim. ²

Attribution graphs and circuit tracing (2025)

The current frontier method replaces part of the model with a trained, more interpretable component — a cross-layer transcoder standing in for the MLPs — and traces the resulting computational graph for a specific prompt. The graph’s nodes are features or neurons and its edges are attributions, giving what Anthropic calls an attribution graph. ¹¹ Circuit Tracing validated the method on an 18-layer model; its companion, On the Biology of a Large Language Model, applied it to Claude 3.5 Haiku across multi-step reasoning, planning in poems, multilingual circuits, addition, medical diagnosis, hallucination, refusal, jailbreaking and chain-of-thought faithfulness. ^{11 3}

The method’s own limitation matters for the dispute: a transcoder is an approximation of the original MLPs, and Cammarata notes that Anthropic’s largest transcoder predicts the same next token as the original only about half the time, with error nodes absorbing the rest. ² Attribution graphs are therefore evidence about a replacement model unless checked against the original — which is exactly what the neuron-level audits claim to do.

The 2026 revision: “Neural Networks Are Already Clean”

Cammarata’s essay — a working hypothesis, with a technical companion paper still forthcoming — argues four things: networks use one learning principle rather than two; ordinary science on them goes much better than expected; understanding compounds; and interpretability may therefore be far more tractable than assumed. ² It is explicitly a reversal: he co-authored Zoom In, proposed the original superposition circuit, and started OpenAI’s automated-interpretability project, which found only about 1.7% of neurons interpretable by its most generous measure in 2023. ²

The founding evidence re-read

• 4e:55 is a V-shape detector, not a cat/car neuron. Its positive pole responds to pointed shapes (fox faces, flames, water jets, shinkansen noses); its negative pole is an O. The whiskers in the famous visualisation come along for the ride: feature visualisation optimises an image to maximise the neuron, and once it finds the cat route it adds everything that route brings, rather than isolating the shared V. ^{2 4}
• The car→dog circuit is a shared-pose circuit. InceptionV1 contains more than a hundred generalised pose neurons that group cars, dogs and other objects by how they face the camera; nothing reads the “hidden” combination, and later layers keep differentiating it into car and pose specialists. ²
• InceptionV1’s illegible last layer was an architectural failure, not superposition. A ConvNeXt of the same parameter count and training data — with residual connections and LayerNorm — has roughly 99% interpretable last-layer neurons. The original 2014 InceptionV1 went from about 5% with the tools of the era to about 50% with better tools, and about 70% when retrained with modern regularisation. ²

The essay generalises: “The story of interpretability, as I see it, has largely been one of finding out that what was under the microscope was better than we thought.” ²

Recursive generalisation

Most of the remaining apparent polysemanticity is a motif he calls recursive generalisation: a neuron grows branches, and then branches of branches, each step locally related to the last. A neuron for “iy” extends to “diy” and then to “do it yourself”, which contains no “iy”; a geology neuron extends to Cesium, a JavaScript geology library; an “enemy” neuron fires on a Russian poem about enemies. ² The claim is that this is ordinary generalisation with a fractal shape, not a second learning principle: the offshoots are rarer, less causally important, and predictable from how the neuron grew. ² The idea was anticipated as “conjoined features” in the 2021 Multimodal Neurons paper, which the essay credits. ²

Interventions that fail to produce packing

If neurons were packing unrelated features, compressing a layer should force visible superposition. The essay’s tests repeatedly find broadening instead:

Manipulation	Result
InceptionV1 layer 3b, 480 → 10 neurons, no bypass	neurons stay interpretable but coarser (“curves & loops” instead of ten curve detectors); 58.35% top-1 / 81.14% top-5 ImageNet
Medium ConvNeXt, final layer 1,024 → 13 neurons	same broadening; no interleaved mixtures
Residual stream narrowed 4× and 8× (to 46 dimensions in the last block)	accuracy falls 72% → 60% / 39%, but the map stays similarly organised; leaks are semantically related, not unrelated
SAE trained on the crushed layer	no clean recovery of the original curve detectors; features are broad and bunched

²

The same pattern appears in the SAE audit: in the last layer, 88% of SAE activity is near-copies of single neurons, removing every dictionary direction that matches no neuron leaves accuracy unchanged, and searching rotations of a layer for the basis that concentrates activity lands back on the neurons. Where SAEs add something, it is loose groupings of existing neurons — spiders in different settings rather than leg-pattern and web neurons. ²

Language-model audits

• Qwen3-32B: roughly 90% of neuron poles have one main job (often with small related offshoots), ~7% have two related jobs, and ~2–4% remain unexplained. The numbers are threshold-dependent; on strong activations in smaller samples he puts the non-arbitrary share nearer 97%. ²
• GPT-2 XL: the model behind the original automated-interpretability study, where fewer than 2% of neurons looked clean, now yields about 70% “roughly one thing” across 500 neuron papers, with only ~5% pristine. ²
• Kimi K3: an attribution graph built from neurons alone in a multi-trillion-parameter model — described as the first published at that scale — resolves a “found wallet” prompt into wallet, cash, owner ID, lost-property and honesty neurons and answers “return”. ²
• Architecture effects: in SwiGLU neurons a double-negative gate × value regime sends activations into the positive pole looking like the negative pole; for about 4% of neurons this dominates the top of the activation list. Ablating these raises most benchmarks but lowers some reasoning benchmarks. ²
• Mixture-of-experts: experts are far more specialised than reported (Qwen3.6 has 10,240 of them), and specialisation acts as an anti-superposition architecture because a packed expert would defeat its own router. ²

Circuit ecology: why networks are clean

The mechanistic explanation is developmental rather than capacity-based. Generalisation and composition win a race: a circuit that already has useful parts and readers captures the next related piece of error before a competitor can grow, so related things consolidate and unrelated things starve (gradient starvation, coherent gradients, circuit racing). ² A neuron that already fires on “diy” is first in line to pick up “do it yourself”; a rare fact cannot support a niche of its own, so it rides existing machinery as a small off-ramp — as when a model taught that an umbrella is a bee reuses existing bee neurons rather than growing a hidden umbrella feature. ² Consolidation into a dedicated neuron is about buildability: a sheltered, findable place for specialists to grow. ²

This yields a testable prediction: the neurons carrying a rare fact should be related to what the fact is about, not unrelated ones chosen because they rarely collide. ²

What “clean” means

“Clean” is not the claim that every mechanism matches a human concept — Cammarata says that would be terrible news and probably impossible. It means the network’s parts are built to be read and built upon by its own downstream readers, so a researcher who learns to read them like a native can follow the organisation. ² The intended picture is a city, not a knot: larger models are easier to navigate because their districts are more differentiated, and the appropriate tool is tethered abstraction — maps that zoom out while every high-level pattern stays connected to the neurons and circuits supporting it. ²

Evidence status (October 2026)

Claim	Status	Basis and what would settle it
4e:55 is a V-shape detector, not cat+car	Well supported, single re-analysis	manual re-inspection, negative pole, circuit connections; independent replication
InceptionV1’s late-layer mess is architectural	Strong	same-size ConvNeXt ≈99% vs 5–70% for InceptionV1
Capacity compression produces broadening, not superposition	Strong in vision	480→10 and 1,024→13 retraining plus SAE checks
SAEs add no fundamental building blocks	Contested	one lab’s audit vs Gorton’s missing curve detectors 10
~90% of Qwen3-32B neurons have one job	Preliminary	threshold-dependent; companion paper awaited
Trillion-parameter models are legible	Anecdotal	one Kimi K3 attribution graph, not yet audited
Superposition does not recur at larger scale	Contested	Nanda: models store more facts than they have neurons; weak activations may carry them 4

Reception

The essay split the field publicly. OpenAI’s Roon called it potentially “a major moment in scientific history” and argued alignment could become a “normal science”; he also cautioned that the results need independent confirmation and that interpretability breakthroughs tend to advance capabilities too. ⁴

Google DeepMind’s Neel Nanda — a founding figure of mech interp — treated it as a useful tool rather than a breakthrough. He accepts that neuron-level analysis is practical and much easier to apply to frontier models, but argues that superposition is still real: language models store far more facts than they have neurons, weak activations plausibly hold much of that information, findings from small image models may not transfer, and the essay defines monosemanticity too loosely. ⁴ Part of the disagreement is definitional: Cammarata has no objection to models representing more concepts than they have dimensions (composition), and Nanda reads him as agreeing on that point; what Cammarata disputes is arbitrary packing of unrelated things. ^{2 4}

Two structural caveats belong on the record: the technical companion paper had not been released at the time of writing, and Goodfire is a commercial interpretability company with an obvious stake in the thesis. ⁴ The essay’s own confidence is about 85% that nearly everything comes from the generalisation process — a claim about the kind of object a network is, not a claim that 85% of mechanisms have been explained. ²

Open questions

• Does it transfer to frontier language models? The strongest evidence is from vision; the language-model audits are preliminary, and attention is barely covered (Cammarata guesses global attention-head names are worth only ~5% marginal explanatory value, and is unsure anyone understands attention well enough to say). ²
• What is the residual 2–4%? Candidates include SwiGLU’s double-negative regime, “semantic residue” around shared shapes, and unresolved architectural artefacts rather than superposition. ²
• Can recursive generalisation be prevented or measured? The essay suggests the offshoots are unimportant enough that a fix may be possible, but does not offer one. ²
• How faithful are attribution graphs? Replacement-model error, error nodes, and the gap between transcoder and original computation remain unresolved. ^{2 11}
• What is the right abstraction? SAEs, transcoders, block-sparse featurizers and meta-models are all candidates; the 2026 essay treats them as views of a legible object rather than necessary replacements, while the featurizer literature argues concepts are genuinely multidimensional and therefore need subspace methods. ^{2 12}
• How should explanations be tested? The essay’s preferred instruments are blind maximum (an agent writes an example from the explanation alone and it must beat the corpus maximum), near misses, and maximally distant concepts; it distrusts single simulation scores. ²

Relation to neuroscience

The essay’s closing section is directly relevant to using networks as model organisms. Cammarata’s circuit-ecology story needs only generalisation, composition and an error signal; if a brain learns, he expects some version of the organising principle to transfer, with polysemanticity in the brain as the obvious counterexample. ² He is equally clear about the humility this implies: having spent a decade on a handful of neurons in models that are far easier to study than a brain, he would discount a cat-and-car reading of a human neuron seen through a few electrodes or a BOLD signal almost to zero. ² That is a caution for the ANN–brain comparison programme represented here by triple-n-dataset: the same lesson — that a mixture seen through a coarse instrument may be an artefact of the instrument — applies with far more force to neural recordings.

See Also

• ai-scientist — autonomous research agents, the workforce the essay proposes for interpretability at scale
• turing-test — the behavioural criterion that mechanistic interpretability is designed to move beyond
• spiking-neural-networks — an alternative substrate whose interpretability is much less studied
• triple-n-dataset — fMRI-guided dense recordings: the ANN-to-brain comparison this debate bears on

References

• Cammarata, N. (2026). Neural Networks Are Already Clean. Goodfire. https://already-clean.goodfire.com/ ²
• Schreiner, M. (2026, 16 October). Is AI interpretability close to a breakthrough? A new essay has researchers divided. The Decoder. https://the-decoder.com/is-ai-interpretability-close-to-a-breakthrough-a-new-essay-has-researchers-divided/ ⁴
• Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. & Carter, S. (2020). Zoom In: An Introduction to Circuits. Distill. https://distill.pub/2020/circuits/zoom-in/ ⁶
• Elhage, N., Hume, T., Olsson, C., et al. (2022). Toy Models of Superposition. Transformer Circuits Thread. https://transformer-circuits.pub/2022/toy_model/index.html ⁸
• Olah, C. (2023). Distributed Representations: Composition & Superposition. Transformer Circuits Thread. https://transformer-circuits.pub/2023/superposition-composition/index.html ⁹
• Bricken, T., Templeton, A., Batson, J., et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html ⁷
• Templeton, A., Conerly, T., Marcus, J., et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/ ⁵
• Ameisen, E., Lindsey, J., Pearce, A., et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/methods.html ¹¹
• Lindsey, J., Gurnee, W., Ameisen, E., et al. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/biology.html ³
• Gorton, L. (2024). The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision. arXiv:2406.03662. https://arxiv.org/abs/2406.03662 ¹⁰
• Sharkey, L., Chughtai, B., Batson, J., et al. (2025). Open Problems in Mechanistic Interpretability. arXiv:2501.16496; TMLR. https://arxiv.org/abs/2501.16496 ¹
• Fel, T., Kowal, M., Jacobs, M., et al. (2026). Uncovering Neural Geometry in Vision Models With Block-Sparse Featurizers. Goodfire. https://www.goodfire.com/research/bsf-vision ¹²

Footnotes

1. raw/papers/sharkey-2025-open-problems-mech-interp-abstract.md ↩ ↩² ↩³

2. raw/articles/cammarata-2026-neural-networks-already-clean.md ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰ ↩²¹ ↩²² ↩²³ ↩²⁴ ↩²⁵ ↩²⁶ ↩²⁷ ↩²⁸ ↩²⁹ ↩³⁰ ↩³¹ ↩³² ↩³³ ↩³⁴ ↩³⁵ ↩³⁶ ↩³⁷ ↩³⁸ ↩³⁹ ↩⁴⁰ ↩⁴¹ ↩⁴² ↩⁴³ ↩⁴⁴ ↩⁴⁵

3. raw/papers/lindsey-2025-biology-of-llm-excerpt.md ↩ ↩² ↩³

4. raw/articles/the-decoder-2026-interpretability-essay-divided.md ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

5. raw/papers/templeton-2024-scaling-monosemanticity-excerpt.md ↩ ↩² ↩³

6. raw/papers/olah-2020-zoom-in-circuits-excerpt.md ↩ ↩² ↩³ ↩⁴

7. raw/papers/bricken-2023-towards-monosemanticity-excerpt.md ↩ ↩² ↩³

8. raw/papers/elhage-2022-toy-models-superposition-excerpt.md ↩ ↩² ↩³ ↩⁴

9. raw/papers/olah-2023-composition-superposition-note.md ↩ ↩² ↩³ ↩⁴

10. raw/papers/gorton-2024-missing-curve-detectors-abstract.md ↩ ↩² ↩³

11. raw/papers/ameisen-2025-circuit-tracing-excerpt.md ↩ ↩² ↩³ ↩⁴

12. raw/papers/fel-2026-block-sparse-featurizers-excerpt.md ↩ ↩²