# Sparse Coding and Autoencoders

arXiv: https://arxiv.org/abs/1708.03735 · HTML: https://mlsanity.com/paper/1708.03735/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2017.

## Built on by (most cited first, 11 of 11)

- [Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models](https://mlsanity.com/paper/2408.00113/) (2024)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [Stable and Interpretable Unrolled Dictionary Learning](https://mlsanity.com/paper/2106.00058/) (2021)
- [Analysis of feature learning in weight-tied autoencoders via the mean field lens](https://mlsanity.com/paper/2102.08373/) (2021)
- [On the convergence of group-sparse autoencoders](https://mlsanity.com/paper/2102.07003/) (2021)
- [Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods](https://mlsanity.com/paper/2212.13468/) (2022)
- [Mechanistic Interpretability with Sparse Autoencoder Neural Operators](https://mlsanity.com/paper/2509.03738/) (2025)
- [Robust Representation Learning for Privacy-Preserving Machine Learning: A Multi-Objective Autoencoder Approach](https://mlsanity.com/paper/2309.04427/) (2023)
- [Effects of sparsity and superposition on loss in simple autoencoders](https://mlsanity.com/paper/2606.18538/) (2026)
- [Sparse-Inductive Generative Adversarial Hashing for Nearest Neighbor Search](https://mlsanity.com/paper/2306.06928/) (2023)
- [Interference Beyond Geometry in Concept Extraction](https://mlsanity.com/paper/2609.35351/) (2026)

## Abstract (arXiv)

In "Dictionary Learning" one tries to recover incoherent matrices A^* ∈ ℝn × h (typically overcomplete and whose columns are assumed to be normalized) and sparse vectors x^* ∈ ℝh with a small support of size hp for some 0 <p < 1 while having access to observations y ∈ ℝn where y = A^*x^*. In this work we undertake a rigorous analysis of whether gradient descent on the squared loss of an autoencoder can solve the dictionary learning problem. The "Autoencoder" architecture we consider is a ℝn rightarrow ℝn mapping with a single ReLU activation layer of size h. Under very mild distributional assumptions on x^*, we prove that the norm of the expected gradient of the standard squared loss function is asymptotically (in sparse code dimension) negligible for all points in a small neighborhood of A^*. This is supported with experimental evidence using synthetic data. We also conduct experiments to suggest that A^* is a local minimum. Along the way we prove that a layer of ReLU gates can be set up to automatically recover the support of the sparse codes. This property holds independent of the loss function. We believe that it could be of independent interest.