mlsanityData to 14 Oct 2026

Empirical Evaluation of Progressive Coding for Sparse Autoencoders

arXiv 2505.00190 ↗First posted 30 Apr 2025cs.LG cs.AIcited by 0

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Sparse autoencoders (SAEs) citepbricken2023monosemanticity,gao2024scalingevaluatingsparseautoencoders rely on dictionary learning to extract interpretable features from neural networks at scale in an unsupervised manner, with applications to representation engineering and information retrieval. SAEs are, however, computationally expensive citeplieberum2024gemmascopeopensparse, especially when multiple SAEs of different sizes are needed. We show that dictionary importance in vanilla SAEs follows a power law. We compare progressive coding based on subset pruning of SAEs -- to jointly training nested SAEs, or so-called em Matryoshka SAEs citepbussmann2024learning,nabeshima2024Matryoshka -- on a language modeling task. We show Matryoshka SAEs exhibit lower reconstruction loss and recaptured language modeling loss, as well as higher representational similarity. Pruned vanilla SAEs are more interpretable, however. We discuss the origins and implications of this trade-off.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 13 of its 26 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

13 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structure10 Oct
  2. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 29 Aug
  3. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning17 May
  4. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control14 May
  5. 2022Toy Models of Superposition21 Sep

Other references · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  5. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 131
  6. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 108
  7. 2014Learning Ordered Representations with Nested Dropoutcited by 84
  8. 2023Do Vision and Language Models Share Concepts? A Vector Space Alignment Studycited by 27

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Identifiable Sparse Autoencoders
  2. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features
  3. 2024Improving Dictionary Learning with Gated Sparse Autoencoders
  4. 2024Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders
  5. 2025SplInterp: Improving our Understanding and Training of Sparse Autoencoders
  6. 2026Are Sparse Autoencoder Benchmarks Reliable?
  7. 2026When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs
  8. 2026Learning Retrieval Models with Sparse Autoencoders