mlsanityData to 7 Oct 2026

Understanding and Robustifying Differentiable Architecture Search

arXiv 1909.09656 ↗First posted 20 Sep 2019cs.LG cs.AI cs.CVcited by 176

Not on the map. Why?

Abstract (arXiv)

Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.

Key facts. Cited by 176 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 66; 2022: 33; 2023: 32; 2024: 19; 2025: 13; 2026: 13.

176 papers.

Sort:

By area:

  1. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  2. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  3. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  4. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  5. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  6. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 88
  7. 2022β-DARTS: Beta-Decay Regularization for Differentiable Architecture Searchcited by 63
  8. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  9. 2021A Survey on Green Deep Learningcited by 52
  10. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  11. 2021Global2Local: Efficient Structure Search for Video Action Segmentationcited by 42
  12. 2022NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easycited by 36
38 more
  1. 2021KNAS: Green Neural Architecture Searchcited by 33
  2. 2021NASI: Label- and Data-agnostic Neural Architecture Search at Initializationcited by 28
  3. 2021Rethinking Graph Neural Architecture Search from Message-passingcited by 26
  4. 2022Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognitioncited by 26
  5. 2022Generalizing Few-Shot NAS with Gradient Matchingcited by 25
  6. 2021EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Searchcited by 24
  7. 2022Shapley-NAS: Discovering Operation Contribution for Neural Architecture Searchcited by 23
  8. 2021CATE: Computation-aware Neural Architecture Encoding with Transformerscited by 21
  9. 2023Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunitiescited by 21
  10. 2021ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxiescited by 20
  11. 2021NAS-Bench-x11 and the Power of Learning Curvescited by 20
  12. 2021Meta Navigator: Search for a Good Adaptation Policy for Few-shot Learningcited by 19
  13. 2022On Redundancy and Diversity in Cell-based Neural Architecture Searchcited by 18
  14. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  15. 2022Efficient Architecture Search for Diverse Taskscited by 18
  16. 2021Towards Accurate and Compact Architectures via Neural Architecture Transformercited by 17
  17. 2021ZARTS: On Zero-order Optimization for Neural Architecture Searchcited by 17
  18. 2021AutoMTL: A Programming Framework for Automating Efficient Multi-Task Learningcited by 17
  19. 2021AutoGEL: An Automated Graph Neural Network with Explicit Link Informationcited by 17
  20. 2022Unifying and Boosting Gradient-Based Training-Free Neural Architecture Searchcited by 17
  21. 2022Learning Where To Look -- Generative NAS is Surprisingly Efficientcited by 15
  22. 2022Lambda-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cellscited by 15
  23. 2023Neural Architecture Design and Robustness: A Datasetcited by 15
  24. 2021Accelerating Neural Architecture Search via Proxy Datacited by 14
  25. 2021Generic Neural Architecture Search via Regressioncited by 14
  26. 2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
  27. 2021Automated Deep Learning: Neural Architecture Search Is Not the Endcited by 13
  28. 2023Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecyclecited by 13
  29. 2021An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Modelscited by 12
  30. 2022DropNAS: Grouped Operation Dropout for Differentiable Architecture Searchcited by 12
  31. 2024Towards Multi-modal Transformers in Federated Learningcited by 12
  32. 2022Neural Architecture Search for Dense Prediction Tasks in Computer Visioncited by 11
  33. 2021iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretizationcited by 10
  34. 2022RF-Next: Efficient Receptive Field Search for Convolutional Neural Networkscited by 10
  35. 2026From Memorization to Creativity: LLM as a Designer of Novel Neural Architecturescited by 10
  36. 2021Zero-Cost Operation Scoring in Differentiable Architecture Searchcited by 9
  37. 2021RANK-NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halvingcited by 9
  38. 2022Less is More: Proxy Datasets in NAS approachescited by 9

50 most cited of 176 All 176 (JSON)