mlsanityData to 7 Oct 2026

Efficient Dictionary Learning with Switch Sparse Autoencoders

arXiv 2410.08201 ↗First posted 10 Oct 2024cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Sparse autoencoders (SAEs) are a recent technique for decomposing neural network activations into human-interpretable features. However, in order for SAEs to identify all features represented in frontier models, it will be necessary to scale them up to very high width, posing a computational challenge. In this work, we introduce Switch Sparse Autoencoders, a novel SAE architecture aimed at reducing the compute cost of training SAEs. Inspired by sparse mixture of experts models, Switch SAEs route activation vectors between smaller "expert" SAEs, enabling SAEs to efficiently scale to many more features. We present experiments comparing Switch SAEs with other SAE architectures, and find that Switch SAEs deliver a substantial Pareto improvement in the reconstruction vs. sparsity frontier for a given fixed training compute budget. We also study the geometry of features across experts, analyze features duplicated across experts, and verify that Switch SAE features are as interpretable as features found by other SAE architectures.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 22 of its 52 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

22 of 52 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2024Mixtral of Expertscited by 1,785
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
10 more
  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  2. 2024Scaling and evaluating sparse autoencoderscited by 530
  3. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  4. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 283
  5. 2013k-Sparse Autoencoderscited by 209
  6. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  7. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  8. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 64
  9. 2019SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systemscited by 46
  10. 2014Zero-bias autoencoders and the benefits of co-adapting featurescited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 23; 2026: 13.

38 papers.

Sort:

By area:

  1. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  2. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 118
  3. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  4. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  5. 2025Route Sparse Autoencoder to Interpret Large Language Modelscited by 25
  6. 2025Incorporating Hierarchical Semantics in Sparse Autoencoder Architecturescited by 12
  7. 2025Probing the Representational Power of Sparse Autoencoders in Vision Modelscited by 9
  8. 2025Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoderscited by 9
  9. 2025The Secret Agenda: LLMs Strategically Lie and Our Current Safety Tools Are Blindcited by 7
  10. 2025Low-Rank Adapting Models for Sparse Autoencoderscited by 6
  11. 2025Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learningcited by 6
  12. 2026DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoderscited by 6
26 more
  1. 2026From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoderscited by 6
  2. 2025Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMscited by 5
  3. 2025Probing Neural Topology of Large Language Modelscited by 4
  4. 2025LLM Interpretability with Identifiable Temporal-Instantaneous Representationcited by 4
  5. 2025Revisiting End-To-End Sparse Autoencoder Training: A Short Finetune Is All You Needcited by 3
  6. 2025Ensembling Sparse Autoencoderscited by 3
  7. 2025Interpreting CFD Surrogates through Sparse Autoencoderscited by 3
  8. 2026Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifoldscited by 3
  9. 2025Are Sparse Autoencoders Useful for Java Function Bug Detection?cited by 2
  10. 2025Insights into a radiology-specialised multimodal large language model with sparse autoencoderscited by 2
  11. 2026Improving Sparse Autoencoder with Dynamic Attentioncited by 2
  12. 2025Time-Aware Feature Selection: Adaptive Temporal Masking for Stable Sparse Autoencoder Trainingcited by 1
  13. 2025Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencodercited by 1
  14. 2026MonoLoss: A Training Objective for Interpretable Monosemantic Representationscited by 1
  15. 2026Sparse Autoencoders as a Steering Basis for Phase Synchronization in Graph-Based CFD Surrogatescited by 1
  16. 2026Beyond Accuracy: Interpreting Topic Representation in Suicide Ideation Detection Modelscited by 1
  17. 2026ICA Lens: Interpreting Language Models Without Training Another Dictionarycited by 1
  18. 2025Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoderscited by 0
  19. 2025Sparsification and Reconstruction from the Perspective of Representation Geometrycited by 0
  20. 2025Concept-SAE: A Controllable and Invertible Concept Interface for Sparse Autoencoderscited by 0
  21. 2025Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimalitycited by 0
  22. 2026Concept Component Analysis: A Principled Approach for Concept Extraction in LLMscited by 0
  23. 2026Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activationscited by 0
  24. 2026Decompose Sparsely Where You Should, Absorb Densely Where You Should Nocited by 0
  25. 2026Scaling Interpretable Transformers with Parity Bottleneck Layerscited by 0
  26. 2026When Trees Are Not Enough: Learning Mixed-Topology Feature Graphs with Adaptive Graph Sparse Autoencoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Sparse Autoencoders, Again?
  2. 2026Sparse Crosscoders for diffing MoEs and Dense models
  3. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features
  4. 2025Interpreting Large Text-to-Image Diffusion Models with Dictionary Learning
  5. 2026SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data
  6. 2026Supervised sparse auto-encoders for interpretable and compositional representations
  7. 2025Interpreting CLIP with Hierarchical Sparse Autoencoders
  8. 2025Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Experts