mlsanityData to 14 Oct 2026

Sparse autoencoder training and evaluation

175 papers4 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: saes, sparse, features, feature, autoencoders

Key facts. Sparse autoencoder training and evaluation: 175 papers on arXiv since 2021 (data through 14 Oct 2026). 84 of its 175 mapped papers (48%) cite “Toy Models of Superposition” (2022, arXiv 2209.10652). 15 new papers 24 Sep–14 Oct.

What’s new

24 Sep – 14 Oct 2026

15 new papers, 24 Sep – 14 Oct; 4 this week.

  1. This week · 4
  2. Complete or Sparse: A Tale of Two Identifiabilities
  3. Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability also fits: Circuits and mechanistic analysis
  4. Readout Scaling Shapes Sparse Feature Recovery
  5. Rescuing Points Trapped Inside Incorrect Decision Regions: A target for Evaluating and Improving Forward-Forward Learning
  6. Earlier · 11
  7. Small area learning
  8. What Do Sparse Autoencoder Feature Discovery Scores Identify? From Activations to Interventions
  9. Inference and learning in sparse autoencoders as natural gradient flow
  10. ScopeSAE: Model-Scope Feature Discovery with Interpretable Layer Selection
  11. The Independence Prior of SAEs Fragments Visual Concepts
  12. ClasSAE: Class-Aligned Sparse Autoencoders via Differentiable Feature-Class Affinity
All 15
  1. Whitening Improves Robustness to Spurious Correlations in Linear Probes
  2. Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs? also fits: Interpreting vision and multimodal models
  3. When Is an SAE Feature Interpretable? A Validation Ladder for EEG Foundation Models
  4. PULSE: Identifying Demonstration-Utility Features with Sparse Autoencoders
  5. Neuralyzing the Trace: Selective Representation-Level Unlearning with Contrastive Sparse Autoencoders also fits: SAE features and their applications

Foundations

Most cited by papers here

  1. Toy Models of Superposition2022cited by 84 here
  2. Learning Multi-Level Features with Matryoshka Sparse Autoencoders2025cited by 42 here
  3. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 22024cited by 40 here
  4. Sparse Autoencoders Do Not Find Canonical Units of Analysis2025cited by 25 here
  5. Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models2025cited by 21 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2022Toy Models of Superpositioncited by 84 here
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 121 here
  3. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 36 here
  4. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 17 here
  5. 2024Scaling and evaluating sparse autoencoderscited by 83 here
  6. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 63 here
  7. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 40 here
  8. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 40 here
  9. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 37 here
  10. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 42 here
  11. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 16 here
13 more it builds on, not in this chain
  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 18 here
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 18 here
  3. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 15 here
  4. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 20 here
  5. 2024Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Modelscited by 16 here
  6. 2024Automatically Interpreting Millions of Features in Large Language Modelscited by 16 here
  7. 2024BatchTopK Sparse Autoencoderscited by 34 here
  8. 2025Sparse Autoencoders Trained on the Same Data Learn Different Featurescited by 24 here
  9. 2025Sparse Autoencoders Do Not Find Canonical Units of Analysiscited by 25 here
  10. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 21 here
  11. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 18 here
  12. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 19 here
  13. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 16 here

The 24 most cited of 246.

Browse

30 of 175, ranked 6–35

  1. 6Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry202519
  2. 7Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning202417
  3. 8From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit202516
  4. 9Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control202415
  5. 10The Origins of Representation Manifolds in Large Language Models202511
  6. 11The Geometry of Concepts: Sparse Autoencoder Feature Structure202411
  7. 12Mathematical Models of Computation in Superposition202410
  8. 13Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations with MDL-SAEs202410
  9. 14Decomposing The Dark Matter of Sparse Autoencoders202410
  10. 15Transcoders Beat Sparse Autoencoders for Interpretability20259
All 30
  1. 16Interpreting Attention Layer Outputs with Sparse Autoencoders20249
  2. 17Do Sparse Autoencoders Capture Concept Manifolds?20268
  3. 18Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs20258
  4. 19The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision20248
  5. 20Efficient Dictionary Learning with Switch Sparse Autoencoders20248
  6. 21Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers20257
  7. 22On the Complexity of Neural Computation in Superposition20247
  8. 23Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders20247
  9. 24Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders20247
  10. 25Feature Hedging: Correlated Features Break Narrow Sparse Autoencoders20256
  11. 26Understanding sparse autoencoder scaling in the presence of feature manifolds20256
  12. 27Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery20246
  13. 28Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small20244
  14. 29Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks20244
  15. 30Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?20263
  16. 31From Data Statistics to Feature Geometry: How Correlations Shape Superposition20263
  17. 32Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality20253
  18. 33Superposition Yields Robust Neural Scaling20253
  19. 34Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders20253
  20. 35Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning20253