# Maximal Sparsity with Deep Networks?

arXiv: https://arxiv.org/abs/1605.01636 · HTML: https://mlsanity.com/paper/1605.01636/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2016.

## Built on by (most cited first, 29 of 29)

- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Equivariant Hypergraph Diffusion Neural Operators](https://mlsanity.com/paper/2207.06680/) (2022)
- [Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection](https://mlsanity.com/paper/2110.05649/) (2021)
- [Hyperparameter Tuning is All You Need for LISTA](https://mlsanity.com/paper/2110.15900/) (2021)
- [Theoretical Perspectives on Deep Learning Methods in Inverse Problems](https://mlsanity.com/paper/2206.14373/) (2022)
- [Stable and Interpretable Unrolled Dictionary Learning](https://mlsanity.com/paper/2106.00058/) (2021)
- [A review and experimental evaluation of deep learning methods for MRI reconstruction](https://mlsanity.com/paper/2109.08618/) (2021)
- [Towards Constituting Mathematical Structures for Learning to Optimize](https://mlsanity.com/paper/2305.18577/) (2023)
- [Generalization Error Bounds for Iterative Recovery Algorithms Unfolded as Neural Networks](https://mlsanity.com/paper/2112.04364/) (2021)
- [DRPCA-Net: Make Robust PCA Great Again for Infrared Small Target Detection](https://mlsanity.com/paper/2507.09541/) (2025)
- [Sparse Bayesian Learning via Stepwise Regression](https://mlsanity.com/paper/2106.06095/) (2021)
- [Deep Algorithm Unrolling for Biomedical Imaging](https://mlsanity.com/paper/2108.06637/) (2021)
- [Hybrid ISTA: Unfolding ISTA With Convergence Guarantees Using Free-Form Deep Neural Networks](https://mlsanity.com/paper/2204.11640/) (2022)
- [PAC-Bayesian Learning of Optimization Algorithms](https://mlsanity.com/paper/2210.11113/) (2022)
- [Learned ISTA with Error-based Thresholding for Adaptive Sparse Coding](https://mlsanity.com/paper/2112.10985/) (2021)
- [Attention-Refined Unrolling for Sparse Sequential micro-Doppler Reconstruction](https://mlsanity.com/paper/2306.14233/) (2023)
- [TpopT: Efficient Trainable Template Optimization on Low-Dimensional Manifolds](https://mlsanity.com/paper/2310.10039/) (2023)
- [SINET: Sparsity-driven Interpretable Neural Network for Underwater Image Enhancement](https://mlsanity.com/paper/2409.01022/) (2024)
- [WARP-LCA: Efficient Convolutional Sparse Coding with Locally Competitive Algorithm](https://mlsanity.com/paper/2410.18794/) (2024)
- [l0-Regularized Sparse Coding-based Interpretable Network for Multi-Modal Image Fusion](https://mlsanity.com/paper/2411.04519/) (2024)
- [Deep greedy unfolding: Sorting out argsorting in greedy sparse recovery algorithms](https://mlsanity.com/paper/2505.15661/) (2025)
- [A Design Space Study for LISTA and Beyond](https://mlsanity.com/paper/2104.04110/) (2021)
- [Unrolled denoising networks provably learn optimal Bayesian inference](https://mlsanity.com/paper/2409.12947/) (2024)
- [A Generalization Result for Convergence in Learning-to-Optimize](https://mlsanity.com/paper/2410.07704/) (2024)
- [Bifurcation Models: Learning Set-Valued Solution Maps with Weight-Tied Dynamics](https://mlsanity.com/paper/2605.07277/) (2026)
- [Learned Interpretable Residual Extragradient ISTA for Sparse Coding](https://mlsanity.com/paper/2106.11970/) (2021)
- [Fast Equivariant Imaging: Accelerating Unsupervised Learning and Model Adaptation via Inexact Splitting](https://mlsanity.com/paper/2507.06764/) (2025)
- [Hyperspectral Unmixing with 3D Convolutional Sparse Coding and Projected Simplex Volume Maximization](https://mlsanity.com/paper/2512.05674/) (2025)
- [Sparse Bayesian Learning Algorithms Revisited: From Learning Majorizers to Structured Algorithmic Learning using Neural Networks](https://mlsanity.com/paper/2604.02513/) (2026)

## Abstract (arXiv)

The iterations of many sparse estimation algorithms are comprised of a fixed linear filter cascaded with a thresholding nonlinearity, which collectively resemble a typical neural network layer. Consequently, a lengthy sequence of algorithm iterations can be viewed as a deep network with shared, hand-crafted layer weights. It is therefore quite natural to examine the degree to which a learned network model might act as a viable surrogate for traditional sparse estimation in domains where ample training data is available. While the possibility of a reduced computational budget is readily apparent when a ceiling is imposed on the number of layers, our work primarily focuses on estimation accuracy. In particular, it is well-known that when a signal dictionary has coherent columns, as quantified by a large RIP constant, then most tractable iterative algorithms are unable to find maximally sparse representations. In contrast, we demonstrate both theoretically and empirically the potential for a trained deep network to recover minimal ℓ0-norm representations in regimes where existing methods fail. The resulting system is deployed on a practical photometric stereo estimation problem, where the goal is to remove sparse outliers that can disrupt the estimation of surface normals from a 3D scene.