Circuit Tracing: Revealing Computational Graphs in Language Models

Deep learning models produce their outputs using a series of transformations distributed across many computational units (artificial “neurons”). The field of mechanistic interpretability seeks to describe these transformations in human-understandable language. To date, our team’s approach has followed a two-step approach. First, we identify features, interpretable building blocks that the model uses in its computations. Second, we describe the processes, or circuits, by which these features interact to produce model outputs.

A natural approach is to use the raw neurons of the model as these building blocks.

1

An alternative approach is to study the roles of gross model components such as entire MLP blocks and attention heads

• Locating and editing factual knowledge in gpt [link]

  K. Meng, D. Bau, A. Andonian, Y. Belinkov.

  arXiv preprint arXiv:2202.05262. 2022.

• Interpretability in the wild: a circuit for indirect object identification in gpt-2 small [link]

  K. Wang, A. Variengien, A. Conmy, B. Shlegeris, J. Steinhardt.

  arXiv preprint arXiv:2211.00593. 2022.

• Towards automated circuit discovery for mechanistic interpretability [PDF]

  A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, A. Garriga-Alonso.

  Advances in Neural Information Processing Systems, Vol 36, pp. 16318–16352. 2023.

[1, 2, 3]

. This approach has identified interesting roles these components play in specific behaviors, but large components play a multitude of unrelated roles across the data distribution, so we seek a more granular decomposition. Using this approach, previous work successfully identified interesting circuits in vision models, built out of neurons that appear to represent meaningful visual concepts

• Thread: Circuits [link]

  N. Cammarata, S. Carter, G. Goh, C. Olah, M. Petrov, L. Schubert, C. Voss, B. Egan, S.K. Lim.

  Distill. 2020.

[4]

. However, model neurons are often polysemantic — representing a mixture of many unrelated concepts. One reason for polysemanticity is thought to be the phenomenon of superposition

• Linear algebraic structure of word senses, with applications to polysemy [PDF]

  S. Arora, Y. Li, Y. Liang, T. Ma, A. Risteski.

  Transactions of the Association for Computational Linguistics, Vol 6, pp. 483–495. MIT Press. 2018.

• Decoding The Thought Vector [link]

  G. Goh. 2016.

• Toy Models of Superposition [HTML]

  N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, C. Olah.

  Transformer Circuits Thread. 2022.

[5, 6, 7]

, in which models must represent more concepts than they have neurons, and thus must “smear” their representation of concepts across many neurons. This mismatch between the network’s basic computational units (neurons) and meaningful concepts has proved a major impediment to progress to the mechanistic agenda, especially in understanding language models.

In recent years, sparse coding models such as sparse autoencoders (SAEs)

• Towards Monosemanticity: Decomposing Language Models With Dictionary Learning [HTML]

  T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J.E. Burke, T. Hume, S. Carter, T. Henighan, C. Olah.

  Transformer Circuits Thread. 2023.

• Sparse Autoencoders Find Highly Interpretable Model Directions [link]

  H. Cunningham, A. Ewart, L. Smith, R. Huben, L. Sharkey.

  arXiv preprint arXiv:2309.08600. 2023.

• Scaling and evaluating sparse autoencoders [link]

  L. Gao, T.D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, J. Wu.

  arXiv preprint arXiv:2406.04093. 2024.

• Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders [link]

  S. Rajamanoharan, T. Lieberum, N. Sonnerat, A. Conmy, V. Varma, J. Kramar, N. Nanda.

  arXiv preprint arXiv:2407.14435. 2024.

[8, 9, 10, 11]

, transcoders

• Transcoders find interpretable LLM feature circuits [PDF]

  J. Dunefsky, P. Chlenski, N. Nanda.

  Advances in Neural Information Processing Systems, Vol 37, pp. 24375–24410. 2025.

• dictionary_learning Github Repository [link]

  S. Marks, A. Karvonen, A. Mueller.

  Github. 2024.

• Predicting Future Activations [link]

  A. Templeton, J. Batson, A. Jermyn, C. Olah.

  2024.

[12, 13, 14]

, and crosscoders

• Sparse Crosscoders for Cross-Layer Features and Model Diffing [HTML]

  J. Lindsey, A. Templeton, J. Marcus, T. Conerly, J. Batson, C. Olah.

  2024.

[15]

have emerged as promising tools for identifying interpretable features represented in superposition. These methods decompose model activations into sparsely active components (“features”),

2

We use ‘feature’ following the tradition of ‘feature detectors’ in neuroscience and ‘feature learning’ in machine learning. Some recent literature uses the term ‘latent,’ which refers to specific vectors in the model’s latent space. We find ‘feature’ better captures the computational role these elements play, making it more appropriate for describing transcoder neurons than SAE decoder vectors. which turn out in many cases to correspond to human-interpretable concepts. While current sparse coding methods are an imperfect way of identifying features (see § 7 Limitations), they produce interpretable enough results that we are motivated to study circuits composed of these features. Several authors have already made promising steps in this direction

• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models [link]

  S. Marks, C. Rager, E.J. Michaud, Y. Belinkov, D. Bau, A. Mueller.

  arXiv preprint arXiv:2403.19647. 2024.

• Transcoders find interpretable LLM feature circuits [PDF]

  J. Dunefsky, P. Chlenski, N. Nanda.

  Advances in Neural Information Processing Systems, Vol 37, pp. 24375–24410. 2025.

• Automatically identifying local and global circuits with linear computation graphs [link]

  X. Ge, F. Zhu, W. Shu, J. Wang, Z. He, X. Qiu.

  arXiv preprint arXiv:2405.13868. 2024.

[16, 12, 17]

.

(Excerpt: introduction and motivation for attribution graphs; full text at source URL.)