A Design Space Study for LISTA and Beyond
Not on the map. Why?
Abstract (arXiv)
In recent years, great success has been witnessed in building problem-specific deep networks from unrolling iterative algorithms, for solving inverse problems and beyond. Unrolling is believed to incorporate the model-based prior with the learning capacity of deep learning. This paper revisits the role of unrolling as a design approach for deep networks: to what extent its resulting special architecture is superior, and can we find better? Using LISTA for sparse recovery as a representative example, we conduct the first thorough design space study for the unrolled models. Among all possible variations, we focus on extensively varying the connectivity patterns and neuron types, leading to a gigantic design space arising from LISTA. To efficiently explore this space and identify top performers, we leverage the emerging tool of neural architecture search (NAS). We carefully examine the searched top architectures in a number of settings, and are able to discover networks that are consistently better than LISTA. We further present more visualization and analysis to "open the black box", and find that the searched top architectures demonstrate highly consistent and potentially transferable patterns. We hope our study to spark more reflections and explorations on how to better mingle model-based optimization prior and data-driven learning.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 43 references that are in the map. First posted on arXiv 8 Apr 2021.
Builds on
28 of 43 references.
References · 28
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Densely Connected Convolutional Networkscited by 6,691
- 2018DARTS: Differentiable Architecture Searchcited by 1,782
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
- 2017Deep Image Priorcited by 1,097
- 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
- 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
- 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 411
- 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
- 2015Conditional Random Fields as Recurrent Neural Networkscited by 213
- 2019FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Searchcited by 129
16 more
- 2018Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholdscited by 83
- 2019Deep Unfolding for Communications Systems: A Survey and Some New Directionscited by 48
- 2018Neural Proximal Gradient Descent for Compressive Imagingcited by 46
- 2016Maximal Sparsity with Deep Networks?cited by 29
- 2019Learning step sizes for unfolded sparse codingcited by 29
- 2018Regularization by architecture: A deep prior approach for inverse problemscited by 26
- 2020Safeguarded Learned Convex Optimizationcited by 24
- 2016D3: Deep Dual-Domain Based Fast Restoration of JPEG-Compressed Imagescited by 22
- 2020Ada-LISTA: Learned Solvers Adaptive to Varying Modelscited by 22
- 2018Learning Deep Gradient Descent Optimization for Image Deconvolutioncited by 18
- 2019Differentiable Linearized ADMMcited by 18
- 2016Tradeoffs between Convergence Speed and Reconstruction Accuracy in Inverse Problemscited by 12
- 2016Understanding Trainable Sparse Coding via Matrix Factorizationcited by 12
- 2016Onsager-corrected deep learning for sparse linear inverse problemscited by 7
- 2018LSALSA: Accelerated Source Separation via Learned Sparse Codingcited by 7
- 2020Theoretical Interpretation of Learned Step Size in Deep-Unfolded Gradient Descentcited by 5
Built on by
Citing papers per year: 2021: 1.
1 paper.
- 2021Hyperparameter Tuning is All You Need for LISTAcited by 22
- 2021Hyperparameter Tuning is All You Need for LISTA29 Oct 2021