mlsanityData to 14 Oct 2026

Scaling sparse feature circuit finding for in-context learning

arXiv 2504.13756 ↗First posted 18 Apr 2025cs.LG cs.AI cs.CLcited by 5

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Sparse autoencoders (SAEs) are a popular tool for interpreting large language model activations, but their utility in addressing open questions in interpretability remains unclear. In this work, we demonstrate their effectiveness by using SAEs to deepen our understanding of the mechanism behind in-context learning (ICL). We identify abstract SAE features that (i) encode the model's knowledge of which task to execute and (ii) whose latent vectors causally induce the task zero-shot. This aligns with prior work showing that ICL is mediated by task vectors. We further demonstrate that these task vectors are well approximated by a sparse sum of SAE latents, including these task-execution features. To explore the ICL mechanism, we adapt the sparse feature circuits methodology of Marks et al. (2024) to work for the much larger Gemma-1 2B model, with 30 times as many parameters, and to the more complex task of ICL. Through circuit finding, we discover task-detecting features with corresponding SAE latents that activate earlier in the prompt, that detect when tasks have been performed. They are causally linked with task-execution features through the attention and MLP sublayers.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 39 of its 60 references that are in the map. First posted on arXiv 18 Apr 2025.

Builds on

39 of 60 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  2. 2024Improving Dictionary Learning with Gated Sparse Autoencoders24 Apr
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 36

  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  3. 2022Toy Models of Superpositioncited by 971
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  5. 2022Transformers learn in-context by gradient descentcited by 732
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  8. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  9. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  10. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  11. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  12. 2023Function Vectors in Large Language Modelscited by 308
24 more
  1. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  2. 2013Direct and Indirect Effectscited by 280
  3. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  4. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  5. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  6. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  7. 2023Localizing Model Behavior with Path Patchingcited by 195
  8. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  9. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  10. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  11. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  12. 2021Equinox: neural networks in JAX via callable PyTrees and filtered transformationscited by 148
  13. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  14. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 67
  15. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  16. 2024Finding Transformer Circuits with Edge Pruningcited by 60
  17. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 50
  18. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  19. 2023Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscited by 45
  20. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  21. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  22. 2024Transformer Circuit Faithfulness Metrics are not Robustcited by 22
  23. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  24. 2024Penzai + Treescope: A Toolkit for Interpreting, Visualizing, and Editing Models As Datacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  2. 2026Evolutionary System Prompt Learning for Reinforcement Learning in LLMscited by 5
  3. 2026What Happens Inside Agent Memory? Circuit Analysis from Emergence to Diagnosiscited by 2
  4. 2025Query Circuits: Explaining How Language Models Answer User Promptscited by 0
  5. 2026Not How Many, But Which: Parameter Placement in Low-Rank Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features
  2. 2024Interpreting Attention Layer Outputs with Sparse Autoencoders
  3. 2026Gradients for Interventions and Activations for Detection: Targeted Feature Learning in Language Models
  4. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
  5. 2025Sparse Autoencoders Trained on the Same Data Learn Different Features
  6. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  7. 2026Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression
  8. 2025Sparse Autoencoder Features for Classifications and Transferability