# New Algorithms for Learning Incoherent and Overcomplete Dictionaries

arXiv: https://arxiv.org/abs/1308.6273 · HTML: https://mlsanity.com/paper/1308.6273/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2013.

## Built on by (most cited first, 22 of 22)

- [Cross-Node Federated Graph Neural Network for Spatio-Temporal Data Modeling](https://mlsanity.com/paper/2106.05223/) (2021)
- [Stable and Interpretable Unrolled Dictionary Learning](https://mlsanity.com/paper/2106.00058/) (2021)
- [Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs](https://mlsanity.com/paper/2505.20254/) (2025)
- [Simple Alternating Minimization Provably Solves Complete Dictionary Learning](https://mlsanity.com/paper/2210.12816/) (2022)
- [Personalized Dictionary Learning for Heterogeneous Datasets](https://mlsanity.com/paper/2305.15311/) (2023)
- [Masked prediction tasks: a parameter identifiability view](https://mlsanity.com/paper/2202.09305/) (2022)
- [Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation models](https://mlsanity.com/paper/2412.16247/) (2024)
- [Dictionary Learning for the Almost-Linear Sparsity Regime](https://mlsanity.com/paper/2210.10855/) (2022)
- [Personalized Convolutional Dictionary Learning of Physiological Time Series](https://mlsanity.com/paper/2503.07687/) (2025)
- [Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretability](https://mlsanity.com/paper/2608.19338/) (2026)
- [Non-Convex Compressed Sensing with Training Data](https://mlsanity.com/paper/2101.08310/) (2021)
- [Deep learning based dictionary learning and tomographic image reconstruction](https://mlsanity.com/paper/2108.11730/) (2021)
- [Convergence of alternating minimisation algorithms for dictionary learning](https://mlsanity.com/paper/2304.01768/) (2023)
- [A Unifying Framework for Unsupervised Concept Extraction](https://mlsanity.com/paper/2604.24936/) (2026)
- [Learning Large Scale Sparse Models](https://mlsanity.com/paper/2301.10958/) (2023)
- [Learning Trees of ℓ0-Minimization Problems](https://mlsanity.com/paper/2302.02548/) (2023)
- [Hiding Data Helps: On the Benefits of Masking for Sparse Coding](https://mlsanity.com/paper/2302.12715/) (2023)
- [Estimation of Models with Limited Data by Leveraging Shared Structure](https://mlsanity.com/paper/2310.02864/) (2023)
- [Efficient Matrix Factorization Via Householder Reflections](https://mlsanity.com/paper/2405.07649/) (2024)
- [Dictionary Learning: The Complexity of Learning Sparse Superposed Features with Feedback](https://mlsanity.com/paper/2502.05407/) (2025)
- [Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs](https://mlsanity.com/paper/2601.20420/) (2026)
- [Iterative Atom Refinement: A Monotonicity Principle for Dictionary Learning](https://mlsanity.com/paper/2609.23812/) (2026)

## Abstract (arXiv)

In sparse recovery we are given a matrix A (the dictionary) and a vector of the form A X where X is sparse, and the goal is to recover X. This is a central notion in signal processing, statistics and machine learning. But in applications such as sparse coding, edge detection, compression and super resolution, the dictionary A is unknown and has to be learned from random examples of the form Y = AX where X is drawn from an appropriate distribution --- this is the dictionary learning problem. In most settings, A is overcomplete: it has more columns than rows. This paper presents a polynomial-time algorithm for learning overcomplete dictionaries; the only previously known algorithm with provable guarantees is the recent work of Spielman, Wang and Wright who gave an algorithm for the full-rank case, which is rarely the case in applications. Our algorithm applies to incoherent dictionaries which have been a central object of study since they were introduced in seminal work of Donoho and Huo. In particular, a dictionary is μ-incoherent if each pair of columns has inner product at most μ / √n. The algorithm makes natural stochastic assumptions about the unknown sparse vector X, which can contain k ≤ c min(√n/μ log n, m1/2 -η) non-zero entries (for any η > 0). This is close to the best k allowable by the best sparse recovery algorithms even if one knows the dictionary A exactly. Moreover, both the running time and sample complexity depend on log 1/ε, where ε is the target accuracy, and so our algorithms converge very quickly to the true dictionary. Our algorithm can also tolerate substantial amounts of noise provided it is incoherent with respect to the dictionary (e.g., Gaussian). In the noisy setting, our running time and sample complexity depend polynomially on 1/ε, and this is necessary.