Is Differentiable Architecture Search truly a One-Shot Method?
Not on the map. Why?
Abstract (arXiv)
Differentiable architecture search (DAS) is a widely researched tool for the discovery of novel architectures, due to its promising results for image classification. The main benefit of DAS is the effectiveness achieved through the weight-sharing one-shot paradigm, which allows efficient architecture search. In this work, we investigate DAS in a systematic case study of inverse problems, which allows us to analyze these potential benefits in a controlled manner. We demonstrate that the success of DAS can be extended from image classification to signal reconstruction, in principle. However, our experiments also expose three fundamental difficulties in the evaluation of DAS-based methods in inverse problems: First, the results show a large variance in all test cases. Second, the final performance is strongly dependent on the hyperparameters of the optimizer. And third, the performance of the weight-sharing architecture used during training does not reflect the final performance of the found architecture well. While the results on image reconstruction confirm the potential of the DAS paradigm, they challenge the common understanding of DAS as a one-shot method.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 34 references that are in the map. First posted on arXiv 12 Aug 2021.
Builds on
28 of 34 references.
References · 28
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
- 2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,352
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
- 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
- 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
- 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
- 2017Progressive Neural Architecture Searchcited by 483
- 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
- 2017MoDL: Model Based Deep Learning Architecture for Inverse Problemscited by 384
- 2017Learning a Variational Network for Reconstruction of Accelerated MRI Datacited by 340
- 2018SNAS: Stochastic Neural Architecture Searchcited by 288
- 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
16 more
- 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
- 2017Learned Primal-dual Reconstructioncited by 251
- 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
- 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
- 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
- 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 175
- 2019Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Searchcited by 129
- 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
- 2019NAS evaluation is frustratingly hardcited by 69
- 2020MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulationcited by 56
- 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
- 2019Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Searchcited by 23
- 2016ATGV-Net: Accurate Depth Super-Resolutioncited by 22
- 2020Smooth Variational Graph Embeddings for Efficient Neural Architecture Searchcited by 12
- 2020Neural Architecture Search as Sparse Supernetcited by 10
- 2020Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernelscited by 7
Built on by
Not cited yet.