mlsanityData to 7 Oct 2026

Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities

arXiv 2307.01998 ↗First posted 5 Jul 2023cs.LG cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

Recently, zero-shot (or training-free) Neural Architecture Search (NAS) approaches have been proposed to liberate NAS from the expensive training process. The key idea behind zero-shot NAS approaches is to design proxies that can predict the accuracy of some given networks without training the network parameters. The proxies proposed so far are usually inspired by recent progress in theoretical understanding of deep learning and have shown great potential on several datasets and NAS benchmarks. This paper aims to comprehensively review and compare the state-of-the-art (SOTA) zero-shot NAS approaches, with an emphasis on their hardware awareness. To this end, we first review the mainstream zero-shot proxies and discuss their theoretical underpinnings. We then compare these zero-shot proxies through large-scale experiments and demonstrate their effectiveness in both hardware-aware and hardware-oblivious NAS scenarios. Finally, we point out several promising ideas to design better proxies. Our source code and the list of related papers are available on https://github.com/SLDGroup/survey-zero-shot-nas.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 91 of its 124 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

91 of 124 references.

References · 91

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2019Searching for MobileNetV3cited by 1,984
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  10. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  11. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  12. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
38 more
  1. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  5. 2017Deep Neural Networks as Gaussian Processescited by 617
  6. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  7. 2019Importance Estimation for Neural Network Pruningcited by 531
  8. 2017Progressive Neural Architecture Searchcited by 483
  9. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  10. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  11. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  12. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  13. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  14. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  15. 2017Large-Scale Evolution of Image Classifierscited by 348
  16. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  17. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  18. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  19. 2016On the Expressive Power of Deep Neural Networkscited by 302
  20. 2018SNAS: Stochastic Neural Architecture Searchcited by 288
  21. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  22. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  23. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  24. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  25. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 253
  26. 2020Neural Architecture Search without Trainingcited by 241
  27. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  28. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  29. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  30. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
  31. 2017Hierarchical Representations for Efficient Architecture Searchcited by 188
  32. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  33. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  34. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  35. 2019Complexity of Linear Regions in Deep Networkscited by 166
  36. 2018Neural Architecture Optimizationcited by 164
  37. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  38. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 7.

21 papers.

Sort:

By area:

  1. 2026Resource-Efficient Iterative LLM-Based NAS with Feedback Memorycited by 7
  2. 2024CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Frameworkcited by 5
  3. 2025L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformerscited by 5
  4. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  5. 2024HyTAS: A Hyperspectral Image Transformer Architecture Search Benchmark and Analysiscited by 3
  6. 2026Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffscited by 3
  7. 2024Zero-Shot NAS via the Suppression of Local Entropy Decreasecited by 2
  8. 2025Towards Efficient Few-shot Graph Neural Architecture Search via Partitioning Gradient Contributioncited by 2
  9. 2024NASH: Neural Architecture and Accelerator Search for Multiplication-Reduced Hybrid Modelscited by 1
  10. 2025SEAL: Searching Expandable Architectures for Incremental Learningcited by 1
  11. 2025Guidelines for the Quality Assessment of Energy-Aware NAS Benchmarkscited by 1
  12. 2025Zero-Shot Neural Architecture Search with Weighted Response Correlationcited by 1
9 more
  1. 2025LLM-NAS: LLM-driven Hardware-Aware Neural Architecture Searchcited by 1
  2. 2025Softlog-Softmax Layers and Divergences Contribute to a Computationally Dependable Ensemble Learningcited by 0
  3. 2025Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Modelscited by 0
  4. 2025Sequencing to Mitigate Catastrophic Forgetting in Continual Learningcited by 0
  5. 2026Efficient Search of Implantable Adaptive Cells for Medical Image Segmentationcited by 0
  6. 2026HERCULES: Hardware-Efficient, Robust, Continual Learning Neural Architecture Searchcited by 0
  7. 2026AutoMCU: Feasibility-First MCU Neural Network Customization via LLM-based Multi-Agent Systemscited by 0
  8. 2026Filling the Pareto-Optimal Front for Affordance Segmentation on Embedded Devices Using RGB-D Camerascited by 0
  9. 2026Bi-EZP: LLM-Guided Bilevel Program Evolution for Ensemble Zero-Cost Proxy Discoverycited by 0