mlsanityData to 14 Oct 2026

SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability

arXiv 2503.09532 ↗First posted 12 Mar 2025cs.LG cs.CLcited by 124

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Sparse autoencoders (SAEs) are a popular technique for interpreting language model activations, and there is extensive recent work on improving SAE effectiveness. However, most prior work evaluates progress using unsupervised proxy metrics with unclear practical relevance. We introduce SAEBench, a comprehensive evaluation suite that measures SAE performance across eight diverse metrics, spanning interpretability, feature disentanglement and practical applications like unlearning. To enable systematic comparison, we open-source a suite of over 200 SAEs across eight recently proposed SAE architectures and training algorithms. Our evaluation reveals that gains on proxy metrics do not reliably translate to better practical performance. For instance, while Matryoshka SAEs slightly underperform on existing proxy metrics, they substantially outperform other architectures on feature disentanglement metrics; moreover, this advantage grows with SAE scale. By providing a standardized framework for measuring progress in SAE development, SAEBench enables researchers to study scaling trends and make nuanced comparisons between different SAE architectures and training methodologies. Our interactive interface enables researchers to flexibly visualize relationships between metrics across hundreds of open-source SAEs at: www.neuronpedia.org/sae-bench

Key facts. Cited by 124 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 17 of its 29 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

17 of 29 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Applying sparse autoencoders to unlearn knowledge in language models25 Oct
  2. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders22 Sep
  3. 2024Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models31 Jul
  4. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  5. 2024Scaling and evaluating sparse autoencoders6 Jun
  6. 2024Improving Dictionary Learning with Gated Sparse Autoencoders24 Apr
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 10

  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  2. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  3. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  4. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  5. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 86
  6. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 79
  7. 2024Efficient Dictionary Learning with Switch Sparse Autoencoderscited by 39
  8. 2024Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Smallcited by 34
  9. 2025Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformerscited by 30
  10. 2024Adaptive Sparse Allocation with Mutual Choice & Feature Choice Sparse Autoencoderscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 44; 2026: 80.

124 papers.

Sort:

By area:

  1. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 143
  2. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 56
  3. 2025Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atomscited by 24
  4. 2025SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMscited by 22
  5. 2025Transferring Linear Features Across Language Models With Model Stitchingcited by 21
  6. 2025Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretabilitycited by 18
  7. 2025Feature Hedging: Correlated Features Break Narrow Sparse Autoencoderscited by 16
  8. 2025Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEscited by 14
  9. 2025On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedycited by 14
  10. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?cited by 14
  11. 2025Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoderscited by 12
  12. 2026SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Datacited by 11
38 more
  1. 2025Measuring and Guiding Monosemanticitycited by 10
  2. 2025Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoderscited by 10
  3. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Featurescited by 9
  4. 2025Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonalitycited by 8
  5. 2025OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Featurescited by 8
  6. 2026AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoderscited by 8
  7. 2026DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoderscited by 7
  8. 2025Resa: Transparent Reasoning Models via SAEscited by 6
  9. 2025Safe-SAIL: Towards a Fine-grained Safety Landscape of Large Language Models via Sparse Autoencoder Interpretation Frameworkcited by 6
  10. 2025LLM Interpretability with Identifiable Temporal-Instantaneous Representationcited by 6
  11. 2025Interpretable and Steerable Concept Bottleneck Sparse Autoencoderscited by 6
  12. 2026From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoderscited by 6
  13. 2026Sparse Autoencoders are Capable LLM Jailbreak Mitigatorscited by 6
  14. 2026Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisationcited by 6
  15. 2025Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoderscited by 5
  16. 2025Evaluating SAE interpretability without explanationscited by 5
  17. 2026Stable and Steerable Sparse Autoencoders with Weight Regularizationcited by 5
  18. 2025Revisiting End-To-End Sparse Autoencoder Training: A Short Finetune Is All You Needcited by 4
  19. 2025Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracycited by 4
  20. 2025Measuring Sparse Autoencoder Feature Sensitivitycited by 4
  21. 2025Memory Retrieval and Consolidation in Large Language Models through Function Tokenscited by 4
  22. 2025AlignSAE: Concept-Aligned Sparse Autoencoderscited by 4
  23. 2025Attribution-Guided Distillation of Matryoshka Sparse Autoencoderscited by 4
  24. 2026Steering LLMs for Culturally Localized Generationcited by 4
  25. 2026Tree SAE: Learning Hierarchical Feature Structures in Sparse Autoencoderscited by 4
  26. 2026Are Sparse Autoencoder Benchmarks Reliable?cited by 4
  27. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selectioncited by 4
  28. 2026ICA Lens: Interpreting Language Models Without Training Another Dictionarycited by 4
  29. 2025Ensembling Sparse Autoencoderscited by 3
  30. 2025Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Modelscited by 3
  31. 2025Mechanistic Interpretability with Sparse Autoencoder Neural Operatorscited by 3
  32. 2025Sparse Autoencoders are Topic Modelscited by 3
  33. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  34. 2026The Rate-Distortion-Polysemanticity Tradeoff in SAEscited by 3
  35. 2025Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.iicited by 2
  36. 2025DB-KSVD: Scalable Alternating Optimization for Disentangling High-Dimensional Embedding Spacescited by 2
  37. 2025CE-Bench: Towards a Reliable Contrastive Evaluation Benchmark of Interpretability of Sparse Autoencoderscited by 2
  38. 2025Re-envisioning Euclid Galaxy Morphology: Identifying and Interpreting Features with Sparse Autoencoderscited by 2

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  2. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models
  3. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility
  4. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
  5. 2025AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations
  6. 2025Sparse Autoencoders Trained on the Same Data Learn Different Features
  7. 2025Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words
  8. 2025Sparse Autoencoder Features for Classifications and Transferability