mlsanityData to 14 Oct 2026

Language Model Circuits Are Sparse in the Neuron Basis

arXiv 2601.22594 ↗First posted 30 Jan 2026cs.CL cs.AIcited by 25

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986). Language model interpretability research has thus turned to techniques which decompose the neuron basis into more interpretable units of model computation, such as sparse autoencoders (SAEs). However, not all neuron-based representations are uninterpretable. For the first time, we empirically show that MLP neurons are as sparse a feature basis as SAEs. We use this finding to develop an end-to-end gradient-based attribution pipeline for circuit tracing on the MLP neuron basis, which surfaces causally effective neurons on a variety of tasks. On a standard subject-verb agreement benchmark (Marks et al., 2025), a circuit of ≈ 102 MLP neurons is enough to control model behaviour. On the multi-hop city-state-capital task from (Lindsey et al., 2025), we find a circuit in which small sets of neurons encode specific latent reasoning steps (e.g. mapping a city to its state), and can be steered to change the model's output. This work thus advances automated interpretability of language models without imposing additional training costs.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 54 of its 76 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

54 of 76 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Weight-sparse transformers have interpretable circuits17 Nov
  2. 2025MIB: A Mechanistic Interpretability Benchmark17 Apr
  3. 2025Open Problems in Mechanistic Interpretability27 Jan
  4. 2024Bilinear MLPs enable weight-based mechanistic interpretability10 Oct
  5. 2024Transcoders Find Interpretable LLM Feature Circuits17 Jun
  6. 2024Automatically Identifying Local and Global Circuits with Linear Computation Graphs22 May
  7. 2023Attribution Patching Outperforms Automated Circuit Discovery16 Oct
  8. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability28 Apr
  9. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability11 Jan
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 44

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  5. 2024Scaling and evaluating sparse autoencoderscited by 553
  6. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  7. 2021Causal Abstractions of Neural Networkscited by 422
  8. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  9. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  10. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  11. 2013Direct and Indirect Effectscited by 280
  12. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
32 more
  1. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  2. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  3. 2023Localizing Model Behavior with Path Patchingcited by 195
  4. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 160
  5. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  6. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 143
  7. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  8. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 120
  9. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  10. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 103
  11. 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
  12. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  13. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 97
  14. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
  15. 2018How Important Is a Neuron?cited by 73
  16. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 61
  17. 2022Probing for the Usage of Grammatical Numbercited by 59
  18. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 46
  19. 2024Designing a Dashboard for Transparency and Control of Conversational AIcited by 46
  20. 2024Decomposing The Dark Matter of Sparse Autoencoderscited by 45
  21. 2025Language Models use Lookbacks to Track Beliefscited by 39
  22. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 34
  23. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 32
  24. 2023A Geometric Notion of Causal Probingcited by 25
  25. 2024Optimal ablation for interpretabilitycited by 25
  26. 2019Explaining and Interpreting LSTMscited by 23
  27. 2018Computationally Efficient Measures of Internal Neuron Importancecited by 20
  28. 2023The System Model and the User Model: Exploring AI Dashboard Designcited by 12
  29. 2023When Language Models Fall in Love: Animacy Processing in Transformer Language Modelscited by 12
  30. 2025Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkitcited by 11
  31. 2025Understanding In-context Learning of Addition via Activation Subspacescited by 8
  32. 2025Modular Arithmetic: Language Models Solve Math Digit by Digitcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 25.

25 papers.

Sort:

By area:

  1. 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
  2. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  3. 2026Latent Planning Emerges with Scalecited by 6
  4. 2026Sparse Reward Subsystem in Large Language Modelscited by 4
  5. 2026Automated Interpretability and Feature Discovery in Language Models with Agentscited by 4
  6. 2026How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuitscited by 2
  7. 2026Sycophancy Suppression Can Impair Rational Updating: Anti-Sycophancy Should Preserve the Ability to Updatecited by 2
  8. 2026Transcoder Adapters for Reasoning-Model Diffingcited by 1
  9. 2026ADAG: Automatically Describing Attribution Graphscited by 1
  10. 2026Targeted Neuron Modulation via Contrastive Pair Searchcited by 1
  11. 2026Turn-Averaged SAEs for Feature Discovery and Long-Context Attributioncited by 1
  12. 2026Through the Looking Glass: Directly Reading and Writing Transformerscited by 1
13 more
  1. 2026Matryoshka attribution: Learning to attribute language model outputs to representations and weightscited by 1
  2. 2026Unveiling the Reasoning Process of Large Language Modelscited by 0
  3. 2026Fast & Faithful Function Vectorscited by 0
  4. 2026Sparsely gated tiny linear expertscited by 0
  5. 2026Distributed Sparse Interventions in Language Modelscited by 0
  6. 2026LLMs Can Annotate Attribution Graphscited by 0
  7. 2026Circuit Condensation: Post-Training that Concentrates a Behavior's Causal Circuitcited by 0
  8. 2026Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influencecited by 0
  9. 2026Understanding and Exploiting Anisotropy in Post-Trainingcited by 0
  10. 2026Active Budget Can Kill Sensitivity: Diagnosing and Repairing TopK Sparse Autoencoder Reliabilitycited by 0
  11. 2026Component and Dimension Sparsity in Transformer Refusal Mechanismscited by 0
  12. 2026Sparse Transformer: a Controlled Testbed for Mechanistic Interpretabilitycited by 0
  13. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scalable Circuit Learning for Interpreting Large Language Models
  2. 2025A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering
  3. 2026Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical Ablation
  4. 2024Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models
  5. 2025Scaling sparse feature circuit finding for in-context learning
  6. 2025Constructing Interpretable Features from Compositional Neuron Groups
  7. 2023N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models
  8. 2024Improving Neuron-level Interpretability with White-box Language Models