# Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities

arXiv: https://arxiv.org/abs/2307.01998 · HTML: https://mlsanity.com/paper/2307.01998/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 91 of its 124 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 30 of 91)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks](https://mlsanity.com/paper/1905.07953/) (2019)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers](https://mlsanity.com/paper/1811.04918/) (2018)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)

## Built on by (most cited first, 21 of 21)

- [Resource-Efficient Iterative LLM-Based NAS with Feedback Memory](https://mlsanity.com/paper/2603.12091/) (2026)
- [CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework](https://mlsanity.com/paper/2406.01414/) (2024)
- [L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformers](https://mlsanity.com/paper/2505.07300/) (2025)
- [Mixed-Precision Quantization for Language Models: Techniques and Prospects](https://mlsanity.com/paper/2510.16805/) (2025)
- [HyTAS: A Hyperspectral Image Transformer Architecture Search Benchmark and Analysis](https://mlsanity.com/paper/2407.16269/) (2024)
- [Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs](https://mlsanity.com/paper/2605.04903/) (2026)
- [Zero-Shot NAS via the Suppression of Local Entropy Decrease](https://mlsanity.com/paper/2411.06236/) (2024)
- [Towards Efficient Few-shot Graph Neural Architecture Search via Partitioning Gradient Contribution](https://mlsanity.com/paper/2506.01231/) (2025)
- [NASH: Neural Architecture and Accelerator Search for Multiplication-Reduced Hybrid Models](https://mlsanity.com/paper/2409.04829/) (2024)
- [SEAL: Searching Expandable Architectures for Incremental Learning](https://mlsanity.com/paper/2505.10457/) (2025)
- [Guidelines for the Quality Assessment of Energy-Aware NAS Benchmarks](https://mlsanity.com/paper/2505.15631/) (2025)
- [Zero-Shot Neural Architecture Search with Weighted Response Correlation](https://mlsanity.com/paper/2507.08841/) (2025)
- [LLM-NAS: LLM-driven Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2510.01472/) (2025)
- [Softlog-Softmax Layers and Divergences Contribute to a Computationally Dependable Ensemble Learning](https://mlsanity.com/paper/2506.04297/) (2025)
- [Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models](https://mlsanity.com/paper/2511.21122/) (2025)
- [Sequencing to Mitigate Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/2512.16871/) (2025)
- [Efficient Search of Implantable Adaptive Cells for Medical Image Segmentation](https://mlsanity.com/paper/2604.14849/) (2026)
- [HERCULES: Hardware-Efficient, Robust, Continual Learning Neural Architecture Search](https://mlsanity.com/paper/2605.04103/) (2026)
- [AutoMCU: Feasibility-First MCU Neural Network Customization via LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2605.21560/) (2026)
- [Filling the Pareto-Optimal Front for Affordance Segmentation on Embedded Devices Using RGB-D Cameras](https://mlsanity.com/paper/2607.28293/) (2026)
- [Bi-EZP: LLM-Guided Bilevel Program Evolution for Ensemble Zero-Cost Proxy Discovery](https://mlsanity.com/paper/2608.21927/) (2026)

## Abstract (arXiv)

Recently, zero-shot (or training-free) Neural Architecture Search (NAS) approaches have been proposed to liberate NAS from the expensive training process. The key idea behind zero-shot NAS approaches is to design proxies that can predict the accuracy of some given networks without training the network parameters. The proxies proposed so far are usually inspired by recent progress in theoretical understanding of deep learning and have shown great potential on several datasets and NAS benchmarks. This paper aims to comprehensively review and compare the state-of-the-art (SOTA) zero-shot NAS approaches, with an emphasis on their hardware awareness. To this end, we first review the mainstream zero-shot proxies and discuss their theoretical underpinnings. We then compare these zero-shot proxies through large-scale experiments and demonstrate their effectiveness in both hardware-aware and hardware-oblivious NAS scenarios. Finally, we point out several promising ideas to design better proxies. Our source code and the list of related papers are available on https://github.com/SLDGroup/survey-zero-shot-nas.