# k-Sparse Autoencoders

arXiv: https://arxiv.org/abs/1312.5663 · HTML: https://mlsanity.com/paper/1312.5663/

## Key facts

- Cited by 215 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2013.

## Built on by (most cited first, 30 of 215)

- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation](https://mlsanity.com/paper/2306.07304/) (2023)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders](https://mlsanity.com/paper/2501.18052/) (2025)
- [Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models](https://mlsanity.com/paper/2504.02821/) (2025)
- [Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry](https://mlsanity.com/paper/2503.01822/) (2025)
- [Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models](https://mlsanity.com/paper/2408.00113/) (2024)
- [LatentQA: Teaching LLMs to Decode Activations Into Natural Language](https://mlsanity.com/paper/2412.08686/) (2024)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [Decomposing The Dark Matter of Sparse Autoencoders](https://mlsanity.com/paper/2410.14670/) (2024)
- [Enhancing Automated Interpretability with Output-Centric Feature Descriptions](https://mlsanity.com/paper/2501.08319/) (2025)
- [Efficient Dictionary Learning with Switch Sparse Autoencoders](https://mlsanity.com/paper/2410.08201/) (2024)
- [LLM Pretraining with Continuous Concepts](https://mlsanity.com/paper/2502.08524/) (2025)
- [Interpreting the linear structure of vision-language model embedding spaces](https://mlsanity.com/paper/2504.11695/) (2025)
- [textitRevelio: Interpreting and leveraging semantic information in diffusion models](https://mlsanity.com/paper/2411.16725/) (2024)
- [Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2410.06981/) (2024)
- [Base Models Know How to Reason, Thinking Models Learn When](https://mlsanity.com/paper/2510.07364/) (2025)
- [Sparse Autoencoders for Hypothesis Generation](https://mlsanity.com/paper/2502.04382/) (2025)
- [Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation](https://mlsanity.com/paper/2503.01776/) (2025)
- [Route Sparse Autoencoder to Interpret Large Language Models](https://mlsanity.com/paper/2503.08200/) (2025)
- [Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders](https://mlsanity.com/paper/2411.01220/) (2024)
- [Interpretable and Testable Vision Features via Sparse Autoencoders](https://mlsanity.com/paper/2502.06755/) (2025)
- [Mixture of Experts Made Intrinsically Interpretable](https://mlsanity.com/paper/2503.07639/) (2025)
- [Geometric Autoencoders -- What You See is What You Decode](https://mlsanity.com/paper/2306.17638/) (2023)
- [Disentangling Dense Embeddings with Sparse Autoencoders](https://mlsanity.com/paper/2408.00657/) (2024)
- [Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations with MDL-SAEs](https://mlsanity.com/paper/2410.11179/) (2024)
- [Regularising Inverse Problems with Generative Machine Learning Models](https://mlsanity.com/paper/2107.11191/) (2021)
- [Emergence and Evolution of Interpretable Concepts in Diffusion Models](https://mlsanity.com/paper/2504.15473/) (2025)
- [Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoders](https://mlsanity.com/paper/2502.15576/) (2025)

## Abstract (arXiv)

Recently, it has been observed that when representations are learnt in a way that encourages sparsity, improved performance is obtained on classification tasks. These methods involve combinations of activation functions, sampling steps and different kinds of penalties. To investigate the effectiveness of sparsity by itself, we propose the k-sparse autoencoder, which is an autoencoder with linear activation function, where in hidden layers only the k highest activities are kept. When applied to the MNIST and NORB datasets, we find that this method achieves better classification results than denoising autoencoders, networks trained with dropout, and RBMs. k-sparse autoencoders are simple to train and the encoding stage is very fast, making them well-suited to large problem sizes, where conventional sparse coding algorithms cannot be applied.