mlsanityData to 7 Oct 2026

Transcoders Find Interpretable LLM Feature Circuits

arXiv 2406.11944 ↗First posted 17 Jun 2024cs.LG cs.CLcited by 186

Not on the map. Why?

Abstract (arXiv)

A key goal in mechanistic interpretability is circuit analysis: finding sparse subgraphs of models corresponding to specific behaviors or capabilities. However, MLP sublayers make fine-grained circuit analysis on transformer-based language models difficult. In particular, interpretable features -- such as those found by sparse autoencoders (SAEs) -- are typically linear combinations of extremely many neurons, each with its own nonlinearity to account for. Circuit analysis in this setting thus either yields intractably large circuits or fails to disentangle local and global behavior. To address this we explore transcoders, which seek to faithfully approximate a densely activating MLP layer with a wider, sparsely-activating MLP layer. We introduce a novel method for using transcoders to perform weights-based circuit analysis through MLP sublayers. The resulting circuits neatly factorize into input-dependent and input-invariant terms. We then successfully train transcoders on language models with 120M, 410M, and 1.4B parameters, and find them to perform at least on par with SAEs in terms of sparsity, faithfulness, and human-interpretability. Finally, we apply transcoders to reverse-engineer unknown circuits in the model, and we obtain novel insights regarding the "greater-than circuit" in GPT2-small. Our results suggest that transcoders can prove effective in decomposing model computations involving MLPs into interpretable circuits. Code is available at https://github.com/jacobdunefsky/transcoder_circuits/.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). Builds on 29 of its 56 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

29 of 56 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  5. 2022In-context Learning and Induction Headscited by 930
  6. 2022Toy Models of Superpositioncited by 928
  7. 2016The Mythos of Model Interpretabilitycited by 716
  8. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  9. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  10. 2021Causal Abstractions of Neural Networkscited by 401
  11. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  12. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
17 more
  1. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  2. 2024How to use and interpret activation patchingcited by 191
  3. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  4. 2023Localizing Model Behavior with Path Patchingcited by 182
  5. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  6. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  7. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  8. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 106
  9. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  10. 2021An Interpretability Illusion for BERTcited by 94
  11. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  12. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  13. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
  14. 2019Correlating neural and symbolic representations of languagecited by 32
  15. 2024Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPTcited by 27
  16. 2024Automatically Identifying Local and Global Circuits with Linear Computation Graphscited by 22
  17. 2023Observable Propagation: Uncovering Feature Vectors in Transformerscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 69; 2026: 108.

186 papers.

Sort:

By area:

  1. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  2. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  3. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  4. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  5. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  6. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  7. 2025Weight-sparse transformers have interpretable circuitscited by 40
  8. 2025Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?cited by 35
  9. 2026Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thoughtcited by 35
  10. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
  11. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 31
  12. 2024Towards Unifying Interpretability and Control: Evaluation via Interventioncited by 28
38 more
  1. 2024Mechanistic?cited by 25
  2. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  3. 2024On the Complexity of Neural Computation in Superpositioncited by 21
  4. 2025Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Videocited by 20
  5. 2024Residual Stream Analysis with Multi-Layer SAEscited by 18
  6. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  7. 2026Language Model Circuits Are Sparse in the Neuron Basiscited by 18
  8. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 15
  9. 2025Mechanistic Interpretability as Statistical Estimation: A Variance Analysiscited by 14
  10. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decodingcited by 13
  11. 2025Analyze Feature Flow to Enhance Interpretation and Steering in Language Modelscited by 12
  12. 2025The Geometry of Self-Verification in a Task-Specific Reasoning Modelcited by 12
  13. 2025Incorporating Hierarchical Semantics in Sparse Autoencoder Architecturescited by 12
  14. 2025Explainability in Practice: A Survey of Explainable NLP Across Various Domainscited by 11
  15. 2025Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoderscited by 11
  16. 2025Understanding sparse autoencoder scaling in the presence of feature manifoldscited by 11
  17. 2025The Dead Salmons of AI Interpretabilitycited by 10
  18. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?cited by 10
  19. 2026Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guaranteescited by 10
  20. 2025Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activationscited by 9
  21. 2025Towards Combinatorial Interpretability of Neural Computationcited by 9
  22. 2025Towards Understanding the Nature of Attention with Low-Rank Sparse Decompositioncited by 9
  23. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  24. 2025Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Modelscited by 8
  25. 2025Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectoriescited by 8
  26. 2025Evolution of Concepts in Language Model Pre-Trainingcited by 8
  27. 2025Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Trainingcited by 8
  28. 2025Towards mechanistic understanding in a data-driven weather model: internal activations reveal interpretable physical featurescited by 8
  29. 2026Quantifying Trust: Financial Risk Management for Trustworthy AI Agentscited by 8
  30. 2025Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learningcited by 7
  31. 2025Tracing Multilingual Representations in LLMs with Cross-Layer Transcoderscited by 7
  32. 2026The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Levelcited by 7
  33. 2025Sparse Feature Coactivation Reveals Causal Semantic Modules in Large Language Modelscited by 6
  34. 2025Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learningcited by 6
  35. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  36. 2025Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoderscited by 6
  37. 2026Circuit Tracing in Vision-Language Models: Understanding the Internal Mechanisms of Multimodal Thinkingcited by 6
  38. 2026Understanding Emergent Misalignment via Feature Superposition Geometrycited by 6

50 most cited of 186 All 186 (JSON)