# Route Sparse Autoencoder to Interpret Large Language Models

arXiv: https://arxiv.org/abs/2503.08200 · HTML: https://mlsanity.com/paper/2503.08200/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [Improving Dictionary Learning with Gated Sparse Autoencoders](https://mlsanity.com/paper/2404.16014/) (2024)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024)
- [Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors](https://mlsanity.com/paper/2103.15949/) (2021)
- [Efficient Dictionary Learning with Switch Sparse Autoencoders](https://mlsanity.com/paper/2410.08201/) (2024)
- [Zero-bias autoencoders and the benefits of co-adapting features](https://mlsanity.com/paper/1402.3337/) (2014)
- [JumpReLU: A Retrofit Defense Strategy for Adversarial Attacks](https://mlsanity.com/paper/1904.03750/) (2019)
- [xSense: Learning Sense-Separated Sparse Representations and Textual Definitions for Explainable Word Sense Networks](https://mlsanity.com/paper/1809.03348/) (2018)
- [Deep Networks for Image Super-Resolution with Sparse Prior](https://mlsanity.com/paper/1507.08905/) (2015)

## Built on by (most cited first, 26 of 26)

- [I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2503.18878/) (2025)
- [Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models](https://mlsanity.com/paper/2605.11887/) (2026)
- [Interpretable Reward Model via Sparse Autoencoder](https://mlsanity.com/paper/2508.08746/) (2025)
- [ProxySPEX: Inference-Efficient Interpretability via Sparse Feature Interactions in LLMs](https://mlsanity.com/paper/2505.17495/) (2025)
- [Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior](https://mlsanity.com/paper/2512.06866/) (2025)
- [SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Models](https://mlsanity.com/paper/2511.20820/) (2025)
- [Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoders](https://mlsanity.com/paper/2512.08892/) (2025)
- [SAFER: Probing Safety in Reward Models with Sparse Autoencoder](https://mlsanity.com/paper/2507.00665/) (2025)
- [Probing Cross-modal Information Hubs in Audio-Visual LLMs](https://mlsanity.com/paper/2605.10815/) (2026)
- [Steering LLMs for Culturally Localized Generation](https://mlsanity.com/paper/2603.23301/) (2026)
- [Multilingual Language Models Encode Script Over Linguistic Structure](https://mlsanity.com/paper/2604.05090/) (2026)
- [To Call or Not to Call: Diagnosing Intrinsic Over-Calling Bias in LLM Agents](https://mlsanity.com/paper/2605.18882/) (2026)
- [DB-KSVD: Scalable Alternating Optimization for Disentangling High-Dimensional Embedding Spaces](https://mlsanity.com/paper/2505.18441/) (2025)
- [Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective](https://mlsanity.com/paper/2505.19815/) (2025)
- [Cross-Layer Discrete Concept Discovery for Interpreting Language Models](https://mlsanity.com/paper/2506.20040/) (2025)
- [SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMs](https://mlsanity.com/paper/2507.14894/) (2025)
- [Can Large Language Models Develop Gambling Addiction?](https://mlsanity.com/paper/2509.22818/) (2025)
- [Breaking the Block: Preserving Data Continuity to Train Superior SAEs for Instruct Models](https://mlsanity.com/paper/2506.07691/) (2025)
- [Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation](https://mlsanity.com/paper/2508.10404/) (2025)
- [Binary Autoencoder for Mechanistic Interpretability of Large Language Models](https://mlsanity.com/paper/2509.20997/) (2025)
- [Graph-Regularized Sparse Autoencoders for LLM Safety Steering](https://mlsanity.com/paper/2512.06655/) (2025)
- [Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMs](https://mlsanity.com/paper/2605.07447/) (2026)
- [Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimality](https://mlsanity.com/paper/2512.10720/) (2025)
- [Uncovering Competency Gaps in Large Language Models and Their Benchmarks](https://mlsanity.com/paper/2512.20638/) (2025)
- [LISE : Listenable Interpretable Speaker Embeddings](https://mlsanity.com/paper/2606.21305/) (2026)
- [ScopeSAE: Model-Scope Feature Discovery with Interpretable Layer Selection](https://mlsanity.com/paper/2610.04905/) (2026)

## Abstract (arXiv)

Mechanistic interpretability of large language models (LLMs) aims to uncover the internal processes of information propagation and reasoning. Sparse autoencoders (SAEs) have demonstrated promise in this domain by extracting interpretable and monosemantic features. However, prior works primarily focus on feature extraction from a single layer, failing to effectively capture activations that span multiple layers. In this paper, we introduce Route Sparse Autoencoder (RouteSAE), a new framework that integrates a routing mechanism with a shared SAE to efficiently extract features from multiple layers. It dynamically assigns weights to activations from different layers, incurring minimal parameter overhead while achieving high interpretability and flexibility for targeted feature manipulation. We evaluate RouteSAE through extensive experiments on Llama-3.2-1B-Instruct. Specifically, under the same sparsity constraint of 64, RouteSAE extracts 22.5% more features than baseline SAEs while achieving a 22.3% higher interpretability score. These results underscore the potential of RouteSAE as a scalable and effective method for LLM interpretability, with applications in feature discovery and model intervention. Our codes are available at https://github.com/swei2001/RouteSAEs.