mlsanityData to 14 Oct 2026

Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders

arXiv 2508.16560 ↗First posted 22 Aug 2025cs.LG cs.AI cs.CLcited by 14

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts. A core SAE training hyperparameter is L0: how many SAE features should fire per token on average. Existing work compares SAE algorithms using sparsity-reconstruction tradeoff plots, implying L0 is a free parameter with no inherently correct value aside from its effect on reconstruction. In this work we study the effect of L0 on SAEs, and show that if L0 is not set correctly, the SAE fails to disentangle the underlying features of the LLM. If L0 is too low, the SAE will mix correlated features to improve reconstruction. If L0 is too high, the SAE finds degenerate solutions that also mix features. Further, we present a proxy metric that can help guide the search for the correct L0 for an SAE on a given training distribution. We show that our method finds the correct L0 in toy models and coincides with peak sparse probing performance in LLM SAEs. We find that most commonly used SAEs have an L0 that is too low. Our work shows that practitioners must set L0 correctly to train SAEs with monosemantic features.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. Builds on 18 of its 31 references that are in the map. First posted on arXiv 22 Aug 2025.

Builds on

18 of 31 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Feature Hedging: Correlated Features Break Narrow Sparse Autoencoders16 May
  2. 2025Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality31 Mar
  3. 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoders21 Mar
  4. 2024Decomposing The Dark Matter of Sparse Autoencoders18 Oct
  5. 2024Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations with MDL-SAEs15 Oct
  6. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 29 Aug
  7. 2022Toy Models of Superposition21 Sep

Other references · 11

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  3. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  4. 2024Scaling and evaluating sparse autoencoderscited by 553
  5. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  6. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  7. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  8. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 160
  9. 2024BatchTopK Sparse Autoencoderscited by 130
  10. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  11. 2016A note on the sample complexity of the Er-SpUD algorithm by Spielman, Wang and Wright for exact recovery of sparsely used dictionariescited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 14.

14 papers.

Sort:

By area:

  1. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?cited by 14
  2. 2026SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Datacited by 11
  3. 2026Learning a Generative Meta-Model of LLM Activationscited by 8
  4. 2026Stable and Steerable Sparse Autoencoders with Weight Regularizationcited by 5
  5. 2026Perplexity Can Miss SAE Feature Damage Under Quantizationcited by 5
  6. 2026The Rate-Distortion-Polysemanticity Tradeoff in SAEscited by 3
  7. 2026Feature Superposition in Neural Networks: From Theory to Practicecited by 1
  8. 2026The Pain Axis: LLMs Represent Self-Directed Harm and Act on Itcited by 1
  9. 2026Building Fast, Evaluating Slow: Pipeline Choices Dominate Autointerpretability Score Variancecited by 0
  10. 2026SharedSAE: One Feature Dictionary Across Language Modelscited by 0
  11. 2026Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?cited by 0
  12. 2026Beyond Token Scale: Chunk-Level Sparse Autoencoders for Reliable Semantic Feature Discoverycited by 0
2 more
  1. 2026A Testable Theory of Atomic Featurescited by 0
  2. 2026Sparse Transformer: a Controlled Testbed for Mechanistic Interpretabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Sparse Autoencoders Trained on the Same Data Learn Different Features
  2. 2026Are Sparse Autoencoder Benchmarks Reliable?
  3. 2024Improving Dictionary Learning with Gated Sparse Autoencoders
  4. 2025SplInterp: Improving our Understanding and Training of Sparse Autoencoders
  5. 2026Improving Robustness In Sparse Autoencoders via Masked Regularization
  6. 2024Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks
  7. 2025Sparse Autoencoders Do Not Find Canonical Units of Analysis
  8. 2024SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoders