mlsanityData to 7 Oct 2026

NASI: Label- and Data-agnostic Neural Architecture Search at Initialization

arXiv 2109.00817 ↗First posted 2 Sep 2021cs.LG cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed a surging interest in Neural Architecture Search (NAS). Various algorithms have been proposed to improve the search efficiency and effectiveness of NAS, i.e., to reduce the search cost and improve the generalization performance of the selected architectures, respectively. However, the search efficiency of these algorithms is severely limited by the need for model training during the search process. To overcome this limitation, we propose a novel NAS algorithm called NAS at Initialization (NASI) that exploits the capability of a Neural Tangent Kernel in being able to characterize the converged performance of candidate architectures at initialization, hence allowing model training to be completely avoided to boost the search efficiency. Besides the improved search efficiency, NASI also achieves competitive search effectiveness on various datasets like CIFAR-10/100 and ImageNet. Further, NASI is shown to be label- and data-agnostic under mild conditions, which guarantees the transferability of architectures selected by our NASI over different datasets.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 48 of its 58 references that are in the map. First posted on arXiv 2 Sep 2021.

Builds on

48 of 58 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  12. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
36 more
  1. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  2. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  3. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  4. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  5. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  6. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  7. 2015Automatic differentiation in machine learning: a surveycited by 846
  8. 2019On the Convergence of Adam and Beyondcited by 834
  9. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  11. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  12. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  13. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  14. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  15. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  16. 2017Progressive Neural Architecture Searchcited by 483
  17. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  18. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  19. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  20. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  21. 2016On the Expressive Power of Deep Neural Networkscited by 302
  22. 2018SNAS: Stochastic Neural Architecture Searchcited by 288
  23. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  24. 2020Neural Architecture Search without Trainingcited by 241
  25. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  26. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  27. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  28. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  29. 2018Neural Architecture Optimizationcited by 164
  30. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  31. 2020DARTS-: Robustly Stepping out of Performance Collapse Without Indicatorscited by 81
  32. 2020NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Searchcited by 64
  33. 2019Understanding Architectures Learnt by Cell-based Neural Architecture Searchcited by 48
  34. 2020Are Labels Necessary for Neural Architecture Search?cited by 37
  35. 2019Efficient Neural Architecture Search via Proximal Iterationscited by 31
  36. 2020Towards NNGP-guided Neural Architecture Searchcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 8; 2024: 8; 2025: 1; 2026: 3.

28 papers.

Sort:

By area:

  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  2. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  3. 2022NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxiescited by 36
  4. 2024PINNACLE: PINN Adaptive ColLocation and Experimental points selectioncited by 31
  5. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  6. 2022Sample-Then-Optimize Batch Neural Thompson Samplingcited by 25
  7. 2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 25
  8. 2024Localized Zeroth-Order Prompt Optimizationcited by 23
  9. 2021ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxiescited by 20
  10. 2022Unifying and Boosting Gradient-Based Training-Free Neural Architecture Searchcited by 17
  11. 2022Generalization Properties of NAS under Activation and Skip Connection Searchcited by 14
  12. 2023DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Networkcited by 14
16 more
  1. 2022Towards Theoretically Inspired Neural Initialization Optimizationcited by 11
  2. 2022Bayesian Optimization under Stochastic Delayed Feedbackcited by 10
  3. 2022FreeREA: Training-Free Evolution-based Architecture Searchcited by 10
  4. 2023β-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Searchcited by 7
  5. 2024Robust NAS under adversarial training: benchmark, theory, and beyondcited by 6
  6. 2023Do Not Train It: A Linear Neural Architecture Search of Graph Neural Networkscited by 5
  7. 2023Training-Free Neural Active Learning with Initialization-Robustness Guaranteescited by 5
  8. 2024ParZC: Parametric Zero-Cost Proxies for Efficient NAScited by 5
  9. 2024Robustifying and Boosting Training-Free Neural Architecture Searchcited by 4
  10. 2023LayerNAS: Neural Architecture Search in Polynomial Complexitycited by 3
  11. 2024pTNAS: Progressive Neural Architecture Search for Tabular Datacited by 2
  12. 2024Hard Work Does Not Always Pay Off: Poisoning Attacks on Neural Architecture Searchcited by 0
  13. 2025Searching Efficient Deep Architectures for Radar Target Detection using Monte-Carlo Tree Searchcited by 0
  14. 2026InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Searchcited by 0
  15. 2026Bi-EZP: LLM-Guided Bilevel Program Evolution for Ensemble Zero-Cost Proxy Discoverycited by 0
  16. 2026Predicting Quantization Price for Selecting PTQ Configurations Before Deploymentcited by 0