mlsanityData to 7 Oct 2026

iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization

arXiv 2108.11014 ↗First posted 25 Aug 2021cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Differentiable ARchiTecture Search (DARTS) uses a continuous relaxation of network representation and dramatically accelerates Neural Architecture Search (NAS) by almost thousands of times in GPU-day. However, the searching process of DARTS is unstable, which suffers severe degradation when training epochs become large, thus limiting its application. In this paper, we claim that this degradation issue is caused by the imbalanced norms between different nodes and the highly correlated outputs from various operations. We then propose an improved version of DARTS, namely iDARTS, to deal with the two problems. In the training phase, it introduces node normalization to maintain the norm balance. In the discretization phase, the continuous architecture is approximated based on the similarity between the outputs of the node and the decorrelated operations rather than the values of the architecture parameters. Extensive evaluation is conducted on CIFAR-10 and ImageNet, and the error rates of 2.25% and 24.7% are reported within 0.2 and 1.9 GPU-day for architecture search respectively, which shows its effectiveness. Additional analysis also reveals that iDARTS has the advantage in robustness and generalization over other DARTS-based counterparts.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 25 Aug 2021.

Builds on

39 of 47 references.

References · 39

  1. 2016Densely Connected Convolutional Networkscited by 6,691
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  7. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  8. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  9. 2016HyperNetworkscited by 1,003
  10. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  11. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  12. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
27 more
  1. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  2. 2017Progressive Neural Architecture Searchcited by 483
  3. 2017Large-Scale Evolution of Image Classifierscited by 348
  4. 2018SNAS: Stochastic Neural Architecture Searchcited by 288
  5. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  6. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  7. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  8. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200
  9. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
  10. 2017Hierarchical Representations for Efficient Architecture Searchcited by 188
  11. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  12. 2018Neural Architecture Optimizationcited by 164
  13. 2017Genetic CNNcited by 124
  14. 2019DARTS+: Improved Differentiable Architecture Search with Early Stoppingcited by 105
  15. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  16. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  17. 2020DSNAS: Direct Neural Architecture Search without Parameter Retrainingcited by 69
  18. 2019BayesNAS: A Bayesian Approach for Neural Architecture Searchcited by 67
  19. 2017A Genetic Programming Approach to Designing Convolutional Neural Network Architecturescited by 59
  20. 2017Simple And Efficient Architecture Search for Convolutional Neural Networkscited by 40
  21. 2018DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architecturescited by 40
  22. 2019XNAS: Neural Architecture Search with Expert Advicecited by 35
  23. 2019ASAP: Architecture Search, Anneal and Prunecited by 29
  24. 2017DeepArchitect: Automatically Designing and Training Deep Architecturescited by 28
  25. 2020GOLD-NAS: Gradual, One-Level, Differentiablecited by 15
  26. 2019Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameterscited by 11
  27. 2018MaskConnect: Connectivity Learning by Gradient Descentcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 1; 2024: 3; 2025: 2.

10 papers.

  1. 2022A Survey on Computationally Efficient Neural Architecture Searchcited by 12
  2. 2025SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Modelscited by 7
  3. 2023IS-DARTS: Stabilizing DARTS through Precise Measurement on Candidate Importancecited by 5
  4. 2024OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strengthcited by 4
  5. 2024A Pairwise Comparison Relation-assisted Multi-objective Evolutionary Neural Architecture Search Method with Multi-population Mechanismcited by 2
  6. 2024SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Searchcited by 1
  7. 2021Exploring Complicated Search Spaces with Interleaving-Free Samplingcited by 0
  8. 2022An Efficient End-to-End 3D Voxel Reconstruction based on Neural Architecture Searchcited by 0
  9. 2022Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategycited by 0
  10. 2025Neural Architecture Search with Mixed Bio-inspired Learning Rulescited by 0