mlsanityData to 14 Oct 2026

Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models

arXiv 2411.00743 ↗First posted 1 Nov 2024cs.LG cs.AI cs.CLcited by 16

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Understanding and mitigating the potential risks associated with foundation models (FMs) hinges on developing effective interpretability methods. Sparse Autoencoders (SAEs) have emerged as a promising tool for disentangling FM representations, but they struggle to capture rare, yet crucial concepts in the data. We introduce Specialized Sparse Autoencoders (SSAEs), designed to illuminate these elusive dark matter features by focusing on specific subdomains. We present a practical recipe for training SSAEs, demonstrating the efficacy of dense retrieval for data selection and the benefits of Tilted Empirical Risk Minimization as a training objective to improve concept recall. Our evaluation of SSAEs on standard metrics, such as downstream perplexity and L0 sparsity, show that they effectively capture subdomain tail concepts, exceeding the capabilities of general-purpose SAEs. We showcase the practical utility of SSAEs in a case study on the Bias in Bios dataset, where SSAEs achieve a 12.5% increase in worst-group classification accuracy when applied to remove spurious gender information. SSAEs provide a powerful new lens for peering into the inner workings of FMs in subdomains.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 39 of its 80 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

39 of 80 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 29 Aug
  2. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning17 May
  3. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control14 May

Other references · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  8. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  9. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  10. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  11. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
24 more
  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2018Disentangling by Factorisingcited by 616
  4. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  5. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  7. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  8. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  9. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  10. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  11. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  12. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
  13. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 179
  14. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  15. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
  16. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 108
  17. 2020Tilted Empirical Risk Minimizationcited by 106
  18. 2021Towards a Theoretical Framework of Out-of-Distribution Generalizationcited by 94
  19. 2013Deep Learning of Representations: Looking Forwardcited by 84
  20. 2014Sparse Modeling for Image and Vision Processingcited by 48
  21. 2023Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Modelscited by 40
  22. 2024The Unreasonable Effectiveness of Easy Training Data for Hard Taskscited by 40
  23. 2023Circuit Breaking: Removing Model Behaviors with Targeted Ablationcited by 27
  24. 2014Zero-bias autoencoders and the benefits of co-adapting featurescited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 8.

16 papers.

Sort:

By area:

  1. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 76
  2. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  3. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  4. 2026AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoderscited by 8
  5. 2025Teach Old SAEs New Domain Tricks with Boostingcited by 5
  6. 2025AlignSAE: Concept-Aligned Sparse Autoencoderscited by 4
  7. 2025Ensembling Sparse Autoencoderscited by 3
  8. 2025Interpreting CFD Surrogates through Sparse Autoencoderscited by 3
  9. 2025Interpreting Language Models Through Concept Descriptions: A Surveycited by 3
  10. 2026Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselinescited by 3
  11. 2025Re-envisioning Euclid Galaxy Morphology: Identifying and Interpreting Features with Sparse Autoencoderscited by 2
  12. 2026Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Featurescited by 1
4 more
  1. 2026Sparse Autoencoders as a Steering Basis for Phase Synchronization in Graph-Based CFD Surrogatescited by 1
  2. 2026From Tokens to Concepts: Leveraging SAE for SPLADEcited by 0
  3. 2026Geometry-Adaptive Explainer for Faithful Dictionary-Based Interpretability under Distribution Shiftcited by 0
  4. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Decomposing The Dark Matter of Sparse Autoencoders
  2. 2025Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse Autoencoders
  3. 2025Don't Forget It! Conditional Sparse Autoencoder Clamping Works for Unlearning
  4. 2026Finding and using interpretable latents in a neutrino foundation model with sparse autoencoders
  5. 2024Disentangling Dense Embeddings with Sparse Autoencoders
  6. 2025Equivariant Sparse Autoencoders: Mechanistic Interpretability of Neural Networks on Symmetric Data
  7. 2026Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?
  8. 2025SplInterp: Improving our Understanding and Training of Sparse Autoencoders