mlsanityData to 7 Oct 2026

Differentiable Architecture Search with Random Features

arXiv 2208.08835 ↗First posted 18 Aug 2022cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Differentiable architecture search (DARTS) has significantly promoted the development of NAS techniques because of its high search efficiency and effectiveness but suffers from performance collapse. In this paper, we make efforts to alleviate the performance collapse problem for DARTS from two aspects. First, we investigate the expressive power of the supernet in DARTS and then derive a new setup of DARTS paradigm with only training BatchNorm. Second, we theoretically find that random features dilute the auxiliary connection role of skip-connection in supernet optimization and enable search algorithm focus on fairer operation selection, thereby solving the performance collapse problem. We instantiate DARTS and PC-DARTS with random features to build an improved version for each named RF-DARTS and RF-PCDARTS respectively. Experimental results show that RF-DARTS obtains 94.36% test accuracy on CIFAR-10 (which is the nearest optimal result in NAS-Bench-201), and achieves the newest state-of-the-art top-1 test error of 24.0% on ImageNet when transferring from CIFAR-10. Moreover, RF-DARTS performs robustly across three datasets (CIFAR-10, CIFAR-100, and SVHN) and four search spaces (S1-S4). Besides, RF-PCDARTS achieves even better results on ImageNet, that is, 23.9% top-1 and 7.1% top-5 test error, surpassing representative methods like single-path, training-free, and partial-channel paradigms directly searched on ImageNet.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 42 of its 53 references that are in the map. First posted on arXiv 18 Aug 2022.

Builds on

42 of 53 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  6. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  9. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  10. 2017Progressive Neural Architecture Searchcited by 483
  11. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  12. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
30 more
  1. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  2. 2018SNAS: Stochastic Neural Architecture Searchcited by 288
  3. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  4. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  5. 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 258
  6. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  7. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  8. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  9. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  10. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  11. 2019Linearized two-layers neural networks in high dimensioncited by 155
  12. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  13. 2019Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Searchcited by 129
  14. 2019DARTS+: Improved Differentiable Architecture Search with Early Stoppingcited by 105
  15. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  16. 2020Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNscited by 103
  17. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  18. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  19. 2020DARTS-: Robustly Stepping out of Performance Collapse Without Indicatorscited by 81
  20. 2019One-Shot Neural Architecture Search via Self-Evaluated Template Networkcited by 69
  21. 2021iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradientscited by 51
  22. 2020Are Labels Necessary for Neural Architecture Search?cited by 37
  23. 2020ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Codingcited by 36
  24. 2021Neural Architecture Search with Random Labelscited by 33
  25. 2021Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Searchcited by 27
  26. 2021BN-NAS: Neural Architecture Search with Batch Normalizationcited by 21
  27. 2019Probabilistic Neural Architecture Searchcited by 20
  28. 2020Cyclic Differentiable Architecture Searchcited by 18
  29. 2019Multinomial Distribution Learning for Effective Neural Architecture Searchcited by 17
  30. 2019Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameterscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 1; 2025: 0; 2026: 2.

4 papers.

  1. 2022Automated Dominative Subspace Mining for Efficient Neural Architecture Searchcited by 3
  2. 2026Bilevel Optimization for Neural Architecture Searchcited by 2
  3. 2024SASE: A Searching Architecture for Squeeze and Excitation Operationscited by 0
  4. 2026Neural Architecture Search for Traffic Prediction: A Survey of Methods, Challenges, and Future Directionscited by 0