# Language Model Circuits Are Sparse in the Neuron Basis

arXiv: https://arxiv.org/abs/2601.22594 · HTML: https://mlsanity.com/paper/2601.22594/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- Builds on 54 of its 76 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 30 of 54)

- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Attribution Patching Outperforms Automated Circuit Discovery](https://mlsanity.com/paper/2310.10348/) (2023)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024)
- [Localizing Model Behavior with Path Patching](https://mlsanity.com/paper/2304.05969/) (2023)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024)
- [Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms](https://mlsanity.com/paper/2403.17806/) (2024)
- [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers](https://mlsanity.com/paper/2402.05602/) (2024)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Interpretability at Scale: Identifying Causal Mechanisms in Alpaca](https://mlsanity.com/paper/2305.08809/) (2023)
- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information](https://mlsanity.com/paper/1808.08079/) (2018)

## Built on by (most cited first, 25 of 25)

- [Verbalizable Representations Form a Global Workspace in Language Models](https://mlsanity.com/paper/2607.15495/) (2026)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Latent Planning Emerges with Scale](https://mlsanity.com/paper/2604.12493/) (2026)
- [Sparse Reward Subsystem in Large Language Models](https://mlsanity.com/paper/2602.00986/) (2026)
- [Automated Interpretability and Feature Discovery in Language Models with Agents](https://mlsanity.com/paper/2605.01555/) (2026)
- [How Much Do Circuits Tell Us? Measuring the Consistency and Specificity of Language Model Circuits](https://mlsanity.com/paper/2605.08348/) (2026)
- [Sycophancy Suppression Can Impair Rational Updating: Anti-Sycophancy Should Preserve the Ability to Update](https://mlsanity.com/paper/2608.26511/) (2026)
- [Transcoder Adapters for Reasoning-Model Diffing](https://mlsanity.com/paper/2602.20904/) (2026)
- [ADAG: Automatically Describing Attribution Graphs](https://mlsanity.com/paper/2604.07615/) (2026)
- [Targeted Neuron Modulation via Contrastive Pair Search](https://mlsanity.com/paper/2605.12290/) (2026)
- [Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution](https://mlsanity.com/paper/2606.28548/) (2026)
- [Through the Looking Glass: Directly Reading and Writing Transformers](https://mlsanity.com/paper/2609.10210/) (2026)
- [Matryoshka attribution: Learning to attribute language model outputs to representations and weights](https://mlsanity.com/paper/2609.25518/) (2026)
- [Unveiling the Reasoning Process of Large Language Models](https://mlsanity.com/paper/2603.29735/) (2026)
- [Fast & Faithful Function Vectors](https://mlsanity.com/paper/2606.05079/) (2026)
- [Sparsely gated tiny linear experts](https://mlsanity.com/paper/2606.07414/) (2026)
- [Distributed Sparse Interventions in Language Models](https://mlsanity.com/paper/2607.07128/) (2026)
- [LLMs Can Annotate Attribution Graphs](https://mlsanity.com/paper/2608.02632/) (2026)
- [Circuit Condensation: Post-Training that Concentrates a Behavior's Causal Circuit](https://mlsanity.com/paper/2608.27254/) (2026)
- [Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence](https://mlsanity.com/paper/2609.18612/) (2026)
- [Understanding and Exploiting Anisotropy in Post-Training](https://mlsanity.com/paper/2609.32792/) (2026)
- [Active Budget Can Kill Sensitivity: Diagnosing and Repairing TopK Sparse Autoencoder Reliability](https://mlsanity.com/paper/2609.37857/) (2026)
- [Component and Dimension Sparsity in Transformer Refusal Mechanisms](https://mlsanity.com/paper/2610.06903/) (2026)
- [Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability](https://mlsanity.com/paper/2610.19907/) (2026)
- [Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility](https://mlsanity.com/paper/2610.20139/) (2026)

## Abstract (arXiv)

The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986). Language model interpretability research has thus turned to techniques which decompose the neuron basis into more interpretable units of model computation, such as sparse autoencoders (SAEs). However, not all neuron-based representations are uninterpretable. For the first time, we empirically show that MLP neurons are as sparse a feature basis as SAEs. We use this finding to develop an end-to-end gradient-based attribution pipeline for circuit tracing on the MLP neuron basis, which surfaces causally effective neurons on a variety of tasks. On a standard subject-verb agreement benchmark (Marks et al., 2025), a circuit of ≈ 102 MLP neurons is enough to control model behaviour. On the multi-hop city-state-capital task from (Lindsey et al., 2025), we find a circuit in which small sets of neurons encode specific latent reasoning steps (e.g. mapping a city to its state), and can be steered to change the model's output. This work thus advances automated interpretability of language models without imposing additional training costs.