# A Design Space Study for LISTA and Beyond

arXiv: https://arxiv.org/abs/2104.04110 · HTML: https://mlsanity.com/paper/2104.04110/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 8 Apr 2021.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Random Search and Reproducibility for Neural Architecture Search](https://mlsanity.com/paper/1902.07638/) (2019)
- [Conditional Random Fields as Recurrent Neural Networks](https://mlsanity.com/paper/1502.03240/) (2015)
- [FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search](https://mlsanity.com/paper/1907.01845/) (2019)
- [Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds](https://mlsanity.com/paper/1808.10038/) (2018)
- [Deep Unfolding for Communications Systems: A Survey and Some New Directions](https://mlsanity.com/paper/1906.05774/) (2019)
- [Neural Proximal Gradient Descent for Compressive Imaging](https://mlsanity.com/paper/1806.03963/) (2018)
- [Learning step sizes for unfolded sparse coding](https://mlsanity.com/paper/1905.11071/) (2019)
- [Maximal Sparsity with Deep Networks?](https://mlsanity.com/paper/1605.01636/) (2016)
- [Regularization by architecture: A deep prior approach for inverse problems](https://mlsanity.com/paper/1812.03889/) (2018)
- [Safeguarded Learned Convex Optimization](https://mlsanity.com/paper/2003.01880/) (2020)
- [D3: Deep Dual-Domain Based Fast Restoration of JPEG-Compressed Images](https://mlsanity.com/paper/1601.04149/) (2016)
- [Ada-LISTA: Learned Solvers Adaptive to Varying Models](https://mlsanity.com/paper/2001.08456/) (2020)
- [Learning Deep Gradient Descent Optimization for Image Deconvolution](https://mlsanity.com/paper/1804.03368/) (2018)
- [Differentiable Linearized ADMM](https://mlsanity.com/paper/1905.06179/) (2019)
- [Tradeoffs between Convergence Speed and Reconstruction Accuracy in Inverse Problems](https://mlsanity.com/paper/1605.09232/) (2016)
- [Understanding Trainable Sparse Coding via Matrix Factorization](https://mlsanity.com/paper/1609.00285/) (2016)
- [Onsager-corrected deep learning for sparse linear inverse problems](https://mlsanity.com/paper/1607.05966/) (2016)
- [LSALSA: Accelerated Source Separation via Learned Sparse Coding](https://mlsanity.com/paper/1802.06875/) (2018)
- [Theoretical Interpretation of Learned Step Size in Deep-Unfolded Gradient Descent](https://mlsanity.com/paper/2001.05142/) (2020)

## Built on by (most cited first, 1 of 1)

- [Hyperparameter Tuning is All You Need for LISTA](https://mlsanity.com/paper/2110.15900/) (2021)

## Abstract (arXiv)

In recent years, great success has been witnessed in building problem-specific deep networks from unrolling iterative algorithms, for solving inverse problems and beyond. Unrolling is believed to incorporate the model-based prior with the learning capacity of deep learning. This paper revisits the role of unrolling as a design approach for deep networks: to what extent its resulting special architecture is superior, and can we find better? Using LISTA for sparse recovery as a representative example, we conduct the first thorough design space study for the unrolled models. Among all possible variations, we focus on extensively varying the connectivity patterns and neuron types, leading to a gigantic design space arising from LISTA. To efficiently explore this space and identify top performers, we leverage the emerging tool of neural architecture search (NAS). We carefully examine the searched top architectures in a number of settings, and are able to discover networks that are consistently better than LISTA. We further present more visualization and analysis to "open the black box", and find that the searched top architectures demonstrate highly consistent and potentially transferable patterns. We hope our study to spark more reflections and explorations on how to better mingle model-based optimization prior and data-driven learning.