# A statistical perspective on algorithm unrolling models for inverse problems

arXiv: https://arxiv.org/abs/2311.06395 · HTML: https://mlsanity.com/paper/2311.06395/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 81 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Builds on (most cited first, 30 of 32)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Learning Deep CNN Denoiser Prior for Image Restoration](https://mlsanity.com/paper/1704.03264/) (2017)
- [MoDL: Model Based Deep Learning Architecture for Inverse Problems](https://mlsanity.com/paper/1712.02862/) (2017)
- [Deep Learning Techniques for Inverse Problems in Imaging](https://mlsanity.com/paper/2005.06001/) (2020)
- [Analyzing Inverse Problems with Invertible Neural Networks](https://mlsanity.com/paper/1808.04730/) (2018)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds](https://mlsanity.com/paper/1808.10038/) (2018)
- [Optimal Rates For Regularization Of Statistical Inverse Learning Problems](https://mlsanity.com/paper/1604.04054/) (2016)
- [One Network to Solve Them All --- Solving Linear Inverse Problems using Deep Projection Models](https://mlsanity.com/paper/1703.09912/) (2017)
- [Posterior Concentration for Sparse Deep Learning](https://mlsanity.com/paper/1803.09138/) (2018)
- [Bayesian linear regression with sparse priors](https://mlsanity.com/paper/1403.0735/) (2014)
- [Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations](https://mlsanity.com/paper/2206.09527/) (2022)
- [Statistical exploration of the Manifold Hypothesis](https://mlsanity.com/paper/2208.11665/) (2022)
- [Learned Convolutional Sparse Coding](https://mlsanity.com/paper/1711.00328/) (2017)
- [Statistical Guarantees for Regularized Neural Networks](https://mlsanity.com/paper/2006.00294/) (2020)
- [Approximation and Estimation for High-Dimensional Deep Learning Networks](https://mlsanity.com/paper/1809.03090/) (2018)
- [On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks](https://mlsanity.com/paper/1806.00701/) (2018)
- [Invert to Learn to Invert](https://mlsanity.com/paper/1911.10914/) (2019)
- [Recent progress in log-concave density estimation](https://mlsanity.com/paper/1709.03154/) (2017)
- [Deep BCD-Net Using Identical Encoding-Decoding CNN Structures for Iterative Image Recovery](https://mlsanity.com/paper/1802.07129/) (2018)
- [Local Linear Convergence of ISTA and FISTA on the LASSO Problem](https://mlsanity.com/paper/1501.02888/) (2015)
- [On Quasi-Newton Forward--Backward Splitting: Proximal Calculus and Convergence](https://mlsanity.com/paper/1801.08691/) (2018)
- [Convergence analysis of Tikhonov regularization for non-linear statistical inverse learning problems](https://mlsanity.com/paper/1902.05404/) (2019)
- [Ensuring Rapid Mixing and Low Bias for Asynchronous Gibbs Sampling](https://mlsanity.com/paper/1602.07415/) (2016)

## Abstract (arXiv)

We consider inverse problems where the conditional distribution of the observation bf y given the latent variable of interest bf x (also known as the forward model) is known, and we have access to a data set in which multiple instances of bf x and bf y are both observed. In this context, algorithm unrolling has become a very popular approach for designing state-of-the-art deep neural network architectures that effectively exploit the forward model. We analyze the statistical complexity of the gradient descent network (GDN), an algorithm unrolling architecture driven by proximal gradient descent. We show that the unrolling depth needed for the optimal statistical performance of GDNs is of order log(n)/log(varrhon-1), where n is the sample size, and varrhon is the convergence rate of the corresponding gradient descent algorithm. We also show that when the negative log-density of the latent variable bf x has a simple proximal operator, then a GDN unrolled at depth D' can solve the inverse problem at the parametric rate O(D'/√n). Our results thus also suggest that algorithm unrolling models are prone to overfitting as the unrolling depth D' increases. We provide several examples to illustrate these results.