# Stable and Interpretable Unrolled Dictionary Learning

arXiv: https://arxiv.org/abs/2106.00058 · HTML: https://mlsanity.com/paper/2106.00058/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 87 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Forward and Reverse Gradient-Based Hyperparameter Optimization](https://mlsanity.com/paper/1703.01785/) (2017)
- [Representer Point Selection for Explaining Deep Neural Networks](https://mlsanity.com/paper/1811.09720/) (2018)
- [Robust and interpretable blind image denoising via bias-free convolutional neural networks](https://mlsanity.com/paper/1906.05478/) (2019)
- [Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds](https://mlsanity.com/paper/1808.10038/) (2018)
- [Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures](https://mlsanity.com/paper/1409.2574/) (2014)
- [Learning to Deblur](https://mlsanity.com/paper/1406.7444/) (2014)
- [Deep Unfolded Robust PCA with Application to Clutter Suppression in Ultrasound](https://mlsanity.com/paper/1811.08252/) (2018)
- [Simple, Efficient, and Neural Algorithms for Sparse Coding](https://mlsanity.com/paper/1503.00778/) (2015)
- [Learning step sizes for unfolded sparse coding](https://mlsanity.com/paper/1905.11071/) (2019)
- [Maximal Sparsity with Deep Networks?](https://mlsanity.com/paper/1605.01636/) (2016)
- [A note on the sample complexity of the Er-SpUD algorithm by Spielman, Wang and Wright for exact recovery of sparsely used dictionaries](https://mlsanity.com/paper/1601.02049/) (2016)
- [Super-efficiency of automatic differentiation for functions defined as a minimum](https://mlsanity.com/paper/2002.03722/) (2020)
- [New Algorithms for Learning Incoherent and Overcomplete Dictionaries](https://mlsanity.com/paper/1308.6273/) (2013)
- [Dictionary Learning and Tensor Decomposition via the Sum-of-Squares Method](https://mlsanity.com/paper/1407.1543/) (2014)
- [Implicit differentiation for fast hyperparameter selection in non-smooth convex learning](https://mlsanity.com/paper/2105.01637/) (2021)
- [Rethinking the CSC Model for Natural Images](https://mlsanity.com/paper/1909.05742/) (2019)
- [Local Linear Convergence of Forward-Backward under Partial Smoothness](https://mlsanity.com/paper/1407.5611/) (2014)
- [Tradeoffs between Convergence Speed and Reconstruction Accuracy in Inverse Problems](https://mlsanity.com/paper/1605.09232/) (2016)
- [Understanding Trainable Sparse Coding via Matrix Factorization](https://mlsanity.com/paper/1609.00285/) (2016)
- [Sparse Coding and Autoencoders](https://mlsanity.com/paper/1708.03735/) (2017)
- [Local Linear Convergence of ISTA and FISTA on the LASSO Problem](https://mlsanity.com/paper/1501.02888/) (2015)
- [A Clustering Approach to Learn Sparsely-Used Overcomplete Dictionaries](https://mlsanity.com/paper/1309.1952/) (2013)
- [Scalable Convolutional Dictionary Learning with Constrained Recurrent Sparse Auto-encoders](https://mlsanity.com/paper/1807.04734/) (2018)
- [NOODL: Provable Online Dictionary Learning and Sparse Coding](https://mlsanity.com/paper/1902.11261/) (2019)
- [Unfolding Neural Networks for Compressive Multichannel Blind Deconvolution](https://mlsanity.com/paper/2010.11391/) (2020)

## Built on by (most cited first, 15 of 15)

- [White-Box Transformers via Sparse Rate Reduction](https://mlsanity.com/paper/2306.01129/) (2023)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [Emergence of Segmentation with Minimalistic White-Box Transformers](https://mlsanity.com/paper/2308.16271/) (2023)
- [Scaling White-Box Transformers for Vision](https://mlsanity.com/paper/2405.20299/) (2024)
- [AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features](https://mlsanity.com/paper/2510.00404/) (2025)
- [Simple Alternating Minimization Provably Solves Complete Dictionary Learning](https://mlsanity.com/paper/2210.12816/) (2022)
- [Learning of Patch-Based Smooth-Plus-Sparse Models for Image Reconstruction](https://mlsanity.com/paper/2412.13070/) (2024)
- [Mechanistic Interpretability with Sparse Autoencoder Neural Operators](https://mlsanity.com/paper/2509.03738/) (2025)
- [Transformers as Intrinsic Optimizers: Forward Inference through the Energy Principle](https://mlsanity.com/paper/2511.00907/) (2025)
- [Unrolled Compressed Blind-Deconvolution](https://mlsanity.com/paper/2209.14165/) (2022)
- [Probabilistic Unrolling: Scalable, Inverse-Free Maximum Likelihood Estimation for Latent Gaussian Models](https://mlsanity.com/paper/2306.03249/) (2023)
- [Evaluating Sparse Autoencoders: From Shallow Design to Matching Pursuit](https://mlsanity.com/paper/2506.05239/) (2025)
- [A Unifying Framework for Unsupervised Concept Extraction](https://mlsanity.com/paper/2604.24936/) (2026)
- [Enhancing Performance of Explainable AI Models with Constrained Concept Refinement](https://mlsanity.com/paper/2502.06775/) (2025)
- [Interference Beyond Geometry in Concept Extraction](https://mlsanity.com/paper/2609.35351/) (2026)

## Abstract (arXiv)

The dictionary learning problem, representing data as a combination of a few atoms, has long stood as a popular method for learning representations in statistics and signal processing. The most popular dictionary learning algorithm alternates between sparse coding and dictionary update steps, and a rich literature has studied its theoretical convergence. The success of dictionary learning relies on access to a "good" initial estimate of the dictionary and the ability of the sparse coding step to provide an unbiased estimate of the code. The growing popularity of unrolled sparse coding networks has led to the empirical finding that backpropagation through such networks performs dictionary learning. We offer the theoretical analysis of these empirical results through PUDLE, a Provable Unrolled Dictionary LEarning method. We provide conditions on the network initialization and data distribution sufficient to recover and preserve the support of the latent code. Additionally, we address two challenges; first, the vanilla unrolled sparse coding computes a biased code estimate, and second, gradients during backpropagated learning can become unstable. We show approaches to reduce the bias of the code estimate in the forward pass, and that of the dictionary estimate in the backward pass. We propose strategies to resolve the learning instability by tuning network parameters and modifying the loss function. Overall, we highlight the impact of loss, unrolling, and backpropagation on convergence. We complement our findings through synthetic and image denoising experiments. Finally, we demonstrate PUDLE's interpretability, a driving factor in designing deep networks based on iterative optimizations, by building a mathematical relation between network weights, its output, and the training set.