mlsanityData to 14 Oct 2026

Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search

arXiv 2304.05405 ↗First posted 11 Apr 2023cs.LG cs.AI cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

In the past few years, Differentiable Neural Architecture Search (DNAS) rapidly imposed itself as the trending approach to automate the discovery of deep neural network architectures. This rise is mainly due to the popularity of DARTS, one of the first major DNAS methods. In contrast with previous works based on Reinforcement Learning or Evolutionary Algorithms, DNAS is faster by several orders of magnitude and uses fewer computational resources. In this comprehensive survey, we focus specifically on DNAS and review recent approaches in this field. Furthermore, we propose a novel challenge-based taxonomy to classify DNAS methods. We also discuss the contributions brought to DNAS in the past few years and its impact on the global NAS field. Finally, we conclude by giving some insights into future research directions for the DNAS field.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 88 of its 143 references that are in the map. First posted on arXiv 11 Apr 2023.

Builds on

88 of 143 references.

References · 88

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  9. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  10. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  11. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  12. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
38 more
  1. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  2. 2017Mixed Precision Trainingcited by 1,156
  3. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  4. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  5. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  8. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  9. 2017Progressive Neural Architecture Searchcited by 487
  10. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  11. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  12. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  13. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  14. 2018SNAS: Stochastic Neural Architecture Searchcited by 289
  15. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  16. 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 260
  17. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 254
  18. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 254
  19. 2020Neural Architecture Search without Trainingcited by 244
  20. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  21. 2019Universally Slimmable Networks and Improved Training Techniquescited by 221
  22. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 188
  23. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 178
  24. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 171
  25. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
  26. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  27. 2019Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Searchcited by 131
  28. 2019DARTS+: Improved Differentiable Architecture Search with Early Stoppingcited by 105
  29. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  30. 2020Explainability in Deep Reinforcement Learningcited by 97
  31. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  32. 2020DARTS-: Robustly Stepping out of Performance Collapse Without Indicatorscited by 81
  33. 2020DSNAS: Direct Neural Architecture Search without Parameter Retrainingcited by 69
  34. 2022β-DARTS: Beta-Decay Regularization for Differentiable Architecture Searchcited by 64
  35. 2016Highway and Residual Networks learn Unrolled Iterative Estimationcited by 52
  36. 2019Densely Connected Search Space for More Flexible Neural Architecture Searchcited by 52
  37. 2021A Survey on Green Deep Learningcited by 52
  38. 2020AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Searchcited by 51

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 1.

5 papers.

  1. 2023Real-Time Semantic Segmentation: A Brief Survey & Comparative Study in Remote Sensingcited by 4
  2. 2024OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strengthcited by 4
  3. 2025Architecture-Aware Minimization (A2M): How to Find Flat Minima in Neural Architecture Searchcited by 2
  4. 2026Align Forward, Adapt Backward: Closing the Discretization Gap in Logic Gate Networkscited by 2
  5. 2025Policy-Driven Transfer Learning in Resource-Limited Animal Monitoringcited by 1