mlsanityData to 14 Oct 2026

Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility

arXiv 2610.20139 ↗First posted 14 Oct 2026cs.CLcited by 0

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Sparse autoencoders (SAEs) have emerged as a promising approach to mechanistic interpretability by decomposing language-model activations into sparse, potentially interpretable features. However, differences in design assumptions, evaluation objectives, and experimental settings complicate conclusions about their effectiveness. We present a critical survey of 138 studies on SAEs in natural language processing. Our contributions are fourfold: (1) we propose a task-grounded taxonomy connecting dictionary design, feature operations, and NLP functions; (2) we examine how sparsity allocation, geometry, data coverage, and inference affect feature recoverability; (3) we assess evidence for semantic interpretability, mechanistic faithfulness, and behavioral utility; and (4) we synthesize findings across multilingual processing, retrieval, reasoning, classification, and behavioral regulation, highlighting the roles of objectives, baselines, supervision, and intervention pipelines. Through comparative analyses of published results, we reconcile apparent disagreements, identify unresolved questions, and propose directions for more rigorous evaluation. An anonymous research register https://anonymous.4open.science/r/sae-research-register-C331/register.html accompanies the survey.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 76 of its 105 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

76 of 105 references.

Closest ancestors · 28

Same direction, newest first

  1. 2026Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression27 May
  2. 2026The Rate-Distortion-Polysemanticity Tradeoff in SAEs14 May
  3. 2026SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training3 Mar
  4. 2026Learning Retrieval Models with Sparse Autoencoders27 Feb
  5. 2025Priors in Time: Missing Inductive Biases for Language Model Interpretability3 Nov
  6. 2025Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability30 Oct
  7. 2025AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations24 Aug
  8. 2025Evaluating Sparse Autoencoders for Monosemantic Representation20 Aug
  9. 2025CRISP: Persistent Concept Unlearning via Sparse Autoencoders19 Aug
  10. 2025SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders11 Aug
  11. 2025On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy19 Jun
  12. 2025Constructing Interpretable Features from Compositional Neuron Groups12 Jun
16 more
  1. 2025Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures1 Jun
  2. 2025SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMs11 Apr
  3. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models7 Mar
  4. 2025Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders5 Mar
  5. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probing23 Feb
  6. 2025Sparse Autoencoder Features for Classifications and Transferability17 Feb
  7. 2025SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models17 Feb
  8. 2024Interpretable Company Similarity with Sparse Autoencoders3 Dec
  9. 2024Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups28 Oct
  10. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders27 Oct
  11. 2024Applying sparse autoencoders to unlearn knowledge in language models25 Oct
  12. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering21 Oct
  13. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders22 Sep
  14. 2024Disentangling Dense Embeddings with Sparse Autoencoders1 Aug
  15. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  16. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 48

  1. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  2. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  3. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  4. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 143
  5. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
  6. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 86
  7. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 80
  8. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 61
  9. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 60
  10. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoderscited by 45
  11. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 41
  12. 2025Interpretable Steering of Large Language Models with Feature Guided Activation Additionscited by 35
36 more
  1. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 31
  2. 2024Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPTcited by 29
  3. 2025Understanding Refusal in Language Models with Sparse Autoencoderscited by 29
  4. 2025Transcoders Beat Sparse Autoencoders for Interpretabilitycited by 25
  5. 2026Language Model Circuits Are Sparse in the Neuron Basiscited by 25
  6. 2025Position: Use Sparse Autoencoders to Discover Unknownscited by 19
  7. 2024Analyzing (In)Abilities of SAEs via Formal Languagescited by 17
  8. 2024Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Modelscited by 16
  9. 2024Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Modelscited by 15
  10. 2026Controllable LLM Reasoning via Sparse Autoencoder-Based Steeringcited by 15
  11. 2025LinguaLens: Towards Interpreting Linguistic Mechanisms of Large Language Models via Sparse Auto-Encodercited by 14
  12. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?cited by 14
  13. 2026Do Sparse Autoencoders Identify Reasoning Features in Language Models?cited by 12
  14. 2026CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmarkcited by 12
  15. 2024Interpret and Control Dense Retrieval with Sparse Latent Featurescited by 10
  16. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddingscited by 10
  17. 2025Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Modelscited by 9
  18. 2025SAFE: A Sparse Autoencoder-Based Framework for Robust Query Enrichment and Hallucination Mitigation in LLMscited by 8
  19. 2025Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonalitycited by 8
  20. 2025Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoderscited by 8
  21. 2026Circuit Tracing in Vision-Language Models: Understanding the Internal Mechanisms of Multimodal Thinkingcited by 8
  22. 2025From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?cited by 7
  23. 2026DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoderscited by 7
  24. 2018Learning and Evaluating Sparse Interpretable Sentence Embeddingscited by 6
  25. 2025Safe-SAIL: Towards a Fine-grained Safety Landscape of Large Language Models via Sparse Autoencoder Interpretation Frameworkcited by 6
  26. 2025Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptationcited by 5
  27. 2026HalluSAE: Detecting Hallucinations in Large Language Models via Sparse Auto-Encoderscited by 5
  28. 2024DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Predictioncited by 4
  29. 2025Sparse Activation Editing for Reliable Instruction Following in Narrativescited by 4
  30. 2026Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraintscited by 4
  31. 2026Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretabilitycited by 4
  32. 2026Multilingual Language Models Encode Script Over Linguistic Structurecited by 3
  33. 2025Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?cited by 2
  34. 2026LANGSAE EDITING: Improving Multilingual Information Retrieval via Post-hoc Language Identity Removalcited by 2
  35. 2026TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignmentcited by 1
  36. 2026REINS: Refusal-Enhanced Inhibitory Steering with Sparse Autoencoder Featurescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words
  2. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability
  3. 2025Evaluating SAE interpretability without explanations
  4. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features
  5. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models
  6. 2025Do Sparse Autoencoders Generalize? A Case Study of Answerability
  7. 2024Automatically Interpreting Millions of Features in Large Language Models
  8. 2026Descriptive Collision in Sparse Autoencoder Auto-Interpretability: When One Explanation Describes Many Features