mlsanityData to 14 Oct 2026

Route Sparse Autoencoder to Interpret Large Language Models

arXiv 2503.08200 ↗First posted 11 Mar 2025cs.LGcited by 26

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Mechanistic interpretability of large language models (LLMs) aims to uncover the internal processes of information propagation and reasoning. Sparse autoencoders (SAEs) have demonstrated promise in this domain by extracting interpretable and monosemantic features. However, prior works primarily focus on feature extraction from a single layer, failing to effectively capture activations that span multiple layers. In this paper, we introduce Route Sparse Autoencoder (RouteSAE), a new framework that integrates a routing mechanism with a shared SAE to efficiently extract features from multiple layers. It dynamically assigns weights to activations from different layers, incurring minimal parameter overhead while achieving high interpretability and flexibility for targeted feature manipulation. We evaluate RouteSAE through extensive experiments on Llama-3.2-1B-Instruct. Specifically, under the same sparsity constraint of 64, RouteSAE extracts 22.5% more features than baseline SAEs while achieving a 22.3% higher interpretability score. These results underscore the potential of RouteSAE as a scalable and effective method for LLM interpretability, with applications in feature discovery and model intervention. Our codes are available at https://github.com/swei2001/RouteSAEs.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 23 of its 39 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

23 of 39 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders27 Oct
  2. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  3. 2024Scaling and evaluating sparse autoencoders6 Jun
  4. 2024Improving Dictionary Learning with Gated Sparse Autoencoders24 Apr
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  7. 2022Toy Models of Superpositioncited by 971
  8. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  9. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  10. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  11. 2013k-Sparse Autoencoderscited by 215
  12. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
6 more
  1. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 108
  2. 2024Efficient Dictionary Learning with Switch Sparse Autoencoderscited by 39
  3. 2014Zero-bias autoencoders and the benefits of co-adapting featurescited by 15
  4. 2019JumpReLU: A Retrofit Defense Strategy for Adversarial Attackscited by 14
  5. 2018xSense: Learning Sense-Separated Sparse Representations and Textual Definitions for Explainable Word Sense Networkscited by 8
  6. 2015Deep Networks for Image Super-Resolution with Sparse Priorcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 8.

26 papers.

Sort:

By area:

  1. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoderscited by 45
  2. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 21
  3. 2025Interpretable Reward Model via Sparse Autoencodercited by 18
  4. 2025ProxySPEX: Inference-Efficient Interpretability via Sparse Feature Interactions in LLMscited by 16
  5. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  6. 2025SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Modelscited by 10
  7. 2025Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoderscited by 7
  8. 2025SAFER: Probing Safety in Reward Models with Sparse Autoencodercited by 6
  9. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMscited by 5
  10. 2026Steering LLMs for Culturally Localized Generationcited by 4
  11. 2026Multilingual Language Models Encode Script Over Linguistic Structurecited by 3
  12. 2026To Call or Not to Call: Diagnosing Intrinsic Over-Calling Bias in LLM Agentscited by 3
14 more
  1. 2025DB-KSVD: Scalable Alternating Optimization for Disentangling High-Dimensional Embedding Spacescited by 2
  2. 2025Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspectivecited by 2
  3. 2025Cross-Layer Discrete Concept Discovery for Interpreting Language Modelscited by 2
  4. 2025SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMscited by 2
  5. 2025Can Large Language Models Develop Gambling Addiction?cited by 2
  6. 2025Breaking the Block: Preserving Data Continuity to Train Superior SAEs for Instruct Modelscited by 1
  7. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  8. 2025Binary Autoencoder for Mechanistic Interpretability of Large Language Modelscited by 1
  9. 2025Graph-Regularized Sparse Autoencoders for LLM Safety Steeringcited by 1
  10. 2026Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMscited by 1
  11. 2025Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimalitycited by 0
  12. 2025Uncovering Competency Gaps in Large Language Models and Their Benchmarkscited by 0
  13. 2026LISE : Listenable Interpretable Speaker Embeddingscited by 0
  14. 2026ScopeSAE: Model-Scope Feature Discovery with Interpretable Layer Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  2. 2026Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions
  3. 2025Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoders
  4. 2025Sparse Shift Autoencoders for Identifying Concepts from Large Language Model Activations
  5. 2026Step-Level Sparse Autoencoder for Reasoning Process Interpretation
  6. 2025Sparse Autoencoder Features for Classifications and Transferability
  7. 2025AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations
  8. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility