# Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility

arXiv: https://arxiv.org/abs/2610.20139 · HTML: https://mlsanity.com/paper/2610.20139/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 76 of its 105 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 30 of 76)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024)
- [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024)
- [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024)
- [Learning Multi-Level Features with Matryoshka Sparse Autoencoders](https://mlsanity.com/paper/2503.17547/) (2025)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control](https://mlsanity.com/paper/2405.08366/) (2024)
- [Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching](https://mlsanity.com/paper/2311.17030/) (2023)
- [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025)
- [Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering](https://mlsanity.com/paper/2410.15999/) (2024)
- [Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry](https://mlsanity.com/paper/2503.01822/) (2025)
- [Interpreting CLIP with Hierarchical Sparse Autoencoders](https://mlsanity.com/paper/2502.20578/) (2025)
- [Applying sparse autoencoders to unlearn knowledge in language models](https://mlsanity.com/paper/2410.19278/) (2024)
- [I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2503.18878/) (2025)
- [Enhancing Automated Interpretability with Output-Centric Feature Descriptions](https://mlsanity.com/paper/2501.08319/) (2025)
- [Interpretable Steering of Large Language Models with Feature Guided Activation Additions](https://mlsanity.com/paper/2501.09929/) (2025)
- [SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models](https://mlsanity.com/paper/2502.11356/) (2025)
- [Do Sparse Autoencoders Capture Concept Manifolds?](https://mlsanity.com/paper/2604.28119/) (2026)
- [Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT](https://mlsanity.com/paper/2402.12201/) (2024)
- [Understanding Refusal in Language Models with Sparse Autoencoders](https://mlsanity.com/paper/2505.23556/) (2025)
- [Transcoders Beat Sparse Autoencoders for Interpretability](https://mlsanity.com/paper/2501.18823/) (2025)
- [Priors in Time: Missing Inductive Biases for Language Model Interpretability](https://mlsanity.com/paper/2511.01836/) (2025)
- [Language Model Circuits Are Sparse in the Neuron Basis](https://mlsanity.com/paper/2601.22594/) (2026)
- [Disentangling Dense Embeddings with Sparse Autoencoders](https://mlsanity.com/paper/2408.00657/) (2024)
- [SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs](https://mlsanity.com/paper/2504.08192/) (2025)
- [Sparse Autoencoder Features for Classifications and Transferability](https://mlsanity.com/paper/2502.11367/) (2025)

## Abstract (arXiv)

Sparse autoencoders (SAEs) have emerged as a promising approach to mechanistic interpretability by decomposing language-model activations into sparse, potentially interpretable features. However, differences in design assumptions, evaluation objectives, and experimental settings complicate conclusions about their effectiveness. We present a critical survey of 138 studies on SAEs in natural language processing. Our contributions are fourfold: (1) we propose a task-grounded taxonomy connecting dictionary design, feature operations, and NLP functions; (2) we examine how sparsity allocation, geometry, data coverage, and inference affect feature recoverability; (3) we assess evidence for semantic interpretability, mechanistic faithfulness, and behavioral utility; and (4) we synthesize findings across multilingual processing, retrieval, reasoning, classification, and behavioral regulation, highlighting the roles of objectives, baselines, supervision, and intervention pipelines. Through comparative analyses of published results, we reconcile apparent disagreements, identify unresolved questions, and propose directions for more rigorous evaluation. An anonymous research register https://anonymous.4open.science/r/sae-research-register-C331/register.html accompanies the survey.