mlsanityData to 7 Oct 2026

Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models

arXiv 2403.19647 ↗First posted 28 Mar 2024cs.LG cs.AI cs.CLcited by 412

Not on the map. Why?

Cited by 23 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce methods for discovering and applying sparse feature circuits. These are causally implicated subnetworks of human-interpretable features for explaining language model behaviors. Circuits identified in prior work consist of polysemantic and difficult-to-interpret units like attention heads or neurons, rendering them unsuitable for many downstream applications. In contrast, sparse feature circuits enable detailed understanding of unanticipated mechanisms. Because they are based on fine-grained units, sparse feature circuits are useful for downstream tasks: We introduce SHIFT, where we improve the generalization of a classifier by ablating features that a human judges to be task-irrelevant. Finally, we demonstrate an entirely unsupervised and scalable interpretability pipeline by discovering thousands of sparse feature circuits for automatically discovered model behaviors.

Key facts. Cited by 412 papers in the map (data through 7 Oct 2026). Builds on 64 of its 86 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

64 of 86 references.

References · 64

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  8. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  11. 2022In-context Learning and Induction Headscited by 930
  12. 2022Toy Models of Superpositioncited by 928
38 more
  1. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  2. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  5. 2018Disentangling by Factorisingcited by 614
  6. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  7. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  9. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  10. 2021Causal Abstractions of Neural Networkscited by 401
  11. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  12. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
  13. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  14. 2018Understanding disentangling in β-VAEcited by 332
  15. 2020Environment Inference for Invariant Learningcited by 306
  16. 2023Function Vectors in Large Language Modelscited by 292
  17. 2023Label-Free Concept Bottleneck Modelscited by 282
  18. 2013Direct and Indirect Effectscited by 276
  19. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  20. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  21. 2013k-Sparse Autoencoderscited by 209
  22. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  23. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  24. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  25. 2021Simple data balancing achieves competitive worst-group-accuracycited by 177
  26. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  27. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  28. 2019Distributionally Robust Language Modelingcited by 142
  29. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  30. 2023The Quantization Model of Neural Scalingcited by 137
  31. 2020Learning from Failure: Training Debiased Classifier from Biased Classifiercited by 133
  32. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  33. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  34. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  35. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  36. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  37. 2022Linear Adversarial Concept Erasurecited by 98
  38. 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 90

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 70; 2025: 162; 2026: 180.

412 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2024Scaling and evaluating sparse autoencoderscited by 530
  3. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  4. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  5. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 283
  6. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  7. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  8. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  9. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 148
  10. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 136
  11. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
38 more
  1. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 118
  2. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  3. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
  4. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  5. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  6. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
  7. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 86
  8. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  9. 2025SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoderscited by 75
  10. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  11. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 64
  12. 2024InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoderscited by 64
  13. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  14. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 60
  15. 2024Applying sparse autoencoders to unlearn knowledge in language modelscited by 58
  16. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  17. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 57
  18. 2024Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Modelscited by 51
  19. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  20. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  21. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  22. 2024Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representationscited by 45
  23. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  24. 2024Interpreting Attention Layer Outputs with Sparse Autoencoderscited by 43
  25. 2024NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internalscited by 43
  26. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoderscited by 43
  27. 2024Decomposing The Dark Matter of Sparse Autoencoderscited by 42
  28. 2025Weight-sparse transformers have interpretable circuitscited by 40
  29. 2024Talking Heads: Understanding Inter-layer Communication in Transformer Language Modelscited by 39
  30. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 39
  31. 2024The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Visioncited by 37
  32. 2024Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Smallcited by 34
  33. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  34. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
  35. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  36. 2025LLM Pretraining with Continuous Conceptscited by 33
  37. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
  38. 2024One-Step is Enough: Sparse Autoencoders for Text-to-Image Diffusion Modelscited by 30

50 most cited of 412 All 412 (JSON)