mlsanityData to 7 Oct 2026

Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search

arXiv 2201.09785 ↗First posted 24 Jan 2022cs.LG cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Neural architecture search (NAS) has gained immense popularity owing to its ability to automate neural architecture design. A number of training-free metrics are recently proposed to realize NAS without training, hence making NAS more scalable. Despite their competitive empirical performances, a unified theoretical understanding of these training-free metrics is lacking. As a consequence, (a) the relationships among these metrics are unclear, (b) there is no theoretical interpretation for their empirical performances, and (c) there may exist untapped potential in existing training-free NAS, which probably can be unveiled through a unified theoretical understanding. To this end, this paper presents a unified theoretical analysis of gradient-based training-free NAS, which allows us to (a) theoretically study their relationships, (b) theoretically guarantee their generalization performances, and (c) exploit our unified theoretical understanding to develop a novel framework named hybrid NAS (HNAS) which consistently boosts training-free NAS in a principled way. Remarkably, HNAS can enjoy the advantages of both training-free (i.e., the superior search efficiency) and training-based (i.e., the remarkable search effectiveness) NAS, which we have demonstrated through extensive experiments.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 52 of its 78 references that are in the map. First posted on arXiv 24 Jan 2022.

Builds on

52 of 78 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2014Going Deeper with Convolutionscited by 4,231
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  8. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  10. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  11. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  12. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
38 more
  1. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  2. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2018Object Detection with Deep Learning: A Reviewcited by 524
  9. 2017Progressive Neural Architecture Searchcited by 483
  10. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  11. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  12. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  13. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  14. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  15. 2018SNAS: Stochastic Neural Architecture Searchcited by 288
  16. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  17. 2020Neural Architecture Search without Trainingcited by 241
  18. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  19. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  20. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  21. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  22. 2018Neural Architecture Optimizationcited by 164
  23. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  24. 2020DARTS-: Robustly Stepping out of Performance Collapse Without Indicatorscited by 81
  25. 2020Federated Bayesian Optimization via Thompson Samplingcited by 77
  26. 2020DrNAS: Dirichlet Neural Architecture Searchcited by 73
  27. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  28. 2019Understanding Architectures Learnt by Cell-based Neural Architecture Searchcited by 48
  29. 2021KNAS: Green Neural Architecture Searchcited by 33
  30. 2021Differentially Private Federated Bayesian Optimization with Distributed Explorationcited by 32
  31. 2019Efficient Neural Architecture Search via Proximal Iterationscited by 31
  32. 2021NASI: Label- and Data-agnostic Neural Architecture Search at Initializationcited by 28
  33. 2019BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budgetcited by 26
  34. 2020Top-k Ranking Bayesian Optimizationcited by 21
  35. 2021GradSign: Model Performance Inference with Theoretical Insightscited by 18
  36. 2020On the Rademacher Complexity of Linear Hypothesis Setscited by 17
  37. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimizationcited by 16
  38. 2019Bayesian Optimization with Binary Auxiliary Informationcited by 15

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 4; 2024: 8; 2025: 2; 2026: 1.

17 papers.

Sort:

By area:

  1. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  2. 2022NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxiescited by 36
  3. 2024PINNACLE: PINN Adaptive ColLocation and Experimental points selectioncited by 31
  4. 2022Sample-Then-Optimize Batch Neural Thompson Samplingcited by 25
  5. 2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 25
  6. 2024Localized Zeroth-Order Prompt Optimizationcited by 23
  7. 2023Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunitiescited by 21
  8. 2023Training-Free Neural Active Learning with Initialization-Robustness Guaranteescited by 5
  9. 2024ParZC: Parametric Zero-Cost Proxies for Efficient NAScited by 5
  10. 2024NeurDB: An AI-powered Autonomous Data Systemcited by 5
  11. 2024Robustifying and Boosting Training-Free Neural Architecture Searchcited by 4
  12. 2024pTNAS: Progressive Neural Architecture Search for Tabular Datacited by 2
5 more
  1. 2023ZiCo-BC: A Bias Corrected Zero-Shot NAS for Vision Taskscited by 1
  2. 2025SAND: A Self-supervised and Adaptive NAS-Driven Framework for Hardware Trojan Detectioncited by 1
  3. 2025Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixturescited by 1
  4. 2024Hard Work Does Not Always Pay Off: Poisoning Attacks on Neural Architecture Searchcited by 0
  5. 2026Predicting Quantization Price for Selecting PTQ Configurations Before Deploymentcited by 0