mlsanityData to 7 Oct 2026

Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics

arXiv 2108.11939 ↗First posted 26 Aug 2021cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation. NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. Recent NAS works start to explore indicators that can predict a network's performance without training. However, they either leveraged limited properties of deep networks, or the benefits of their training-free indicators are not applied to more extensive search methods. By rigorous correlation analysis, we present a unified framework to understand and accelerate NAS, by disentangling "TEG" characteristics of searched networks - Trainability, Expressivity, Generalization - all assessed in a training-free manner. The TEG indicators could be scaled up and integrated with various NAS search methods, including both supernet and single-path approaches. Extensive studies validate the effective and efficient guidance from our TEG-NAS framework, leading to both improved search accuracy and over 56% reduction in search time cost. Moreover, we visualize search trajectories on three landscapes of "TEG" characteristics, observing that while a good local minimum is easier to find on NAS-Bench-201 given its simple topology, balancing "TEG" characteristics is much harder on the DARTS search space due to its complex landscape geometry. Our code is available at https://github.com/VITA-Group/TEGNAS.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 81 of its 106 references that are in the map. First posted on arXiv 26 Aug 2021.

Builds on

81 of 106 references.

References · 81

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  12. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
38 more
  1. 2020Designing Network Design Spacescited by 954
  2. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  3. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  4. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2018On Lazy Training in Differentiable Programmingcited by 671
  7. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  10. 2017Progressive Neural Architecture Searchcited by 483
  11. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  12. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  13. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  14. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  15. 2016On the Expressive Power of Deep Neural Networkscited by 302
  16. 2018SNAS: Stochastic Neural Architecture Searchcited by 288
  17. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  18. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  19. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  20. 2020Neural Architecture Search without Trainingcited by 241
  21. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  22. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  23. 2017SMASH: One-Shot Model Architecture Search through HyperNetworkscited by 200
  24. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  25. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
  26. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  27. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  28. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  29. 2019Complexity of Linear Regions in Deep Networkscited by 166
  30. 2016Deep Pyramidal Residual Networkscited by 163
  31. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  32. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  33. 2019Evaluating the Search Phase of Neural Architecture Searchcited by 120
  34. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  35. 2020Predicting Neural Network Accuracy from Weightscited by 107
  36. 2019DARTS+: Improved Differentiable Architecture Search with Early Stoppingcited by 105
  37. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  38. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 0; 2025: 2.

7 papers.

  1. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  2. 2023Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunitiescited by 21
  3. 2022Auto-scaling Vision Transformers without Trainingcited by 15
  4. 2023Efficient Model Adaptation for Continual Learning at the Edgecited by 3
  5. 2022The BUTTER Zone: An Empirical Study of Training Dynamics in Fully Connected Neural Networkscited by 1
  6. 2025Interpretable Recognition of Fused Magnesium Furnace Working Conditions with Deep Convolutional Stochastic Configuration Networkscited by 0
  7. 2025Searching on a Budget: HW-NAS with 10 Latency Probescited by 0