# AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search

arXiv: https://arxiv.org/abs/2403.19232 · HTML: https://mlsanity.com/paper/2403.19232/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 80 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Builds on (most cited first, 30 of 61)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)

## Built on by (most cited first, 28 of 28)

- [Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback](https://mlsanity.com/paper/2602.12612/) (2026)
- [L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformers](https://mlsanity.com/paper/2505.07300/) (2025)
- [Subnet-Aware Dynamic Supernet Training for Neural Architecture Search](https://mlsanity.com/paper/2503.10740/) (2025)
- [Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights](https://mlsanity.com/paper/2502.04975/) (2025)
- [Towards certifiable AI in aviation: landscape, challenges, and opportunities](https://mlsanity.com/paper/2409.08666/) (2024)
- [Zero-Shot NAS via the Suppression of Local Entropy Decrease](https://mlsanity.com/paper/2411.06236/) (2024)
- [CARL: Causality-guided Architecture Representation Learning for an Interpretable Performance Predictor](https://mlsanity.com/paper/2506.04001/) (2025)
- [Bilevel Optimization for Neural Architecture Search](https://mlsanity.com/paper/2606.29582/) (2026)
- [GreenMachine: Automatic Design of Zero-Cost Proxies for Energy-Efficient NAS](https://mlsanity.com/paper/2411.15290/) (2024)
- [Zero-Shot Neural Architecture Search with Weighted Response Correlation](https://mlsanity.com/paper/2507.08841/) (2025)
- [QuProFS: An Evolutionary Training-free Approach to Efficient Quantum Feature Map Search](https://mlsanity.com/paper/2508.07104/) (2025)
- [Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvature](https://mlsanity.com/paper/2508.12977/) (2025)
- [LLM-NAS: LLM-driven Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2510.01472/) (2025)
- [Universal Neural Architecture Space: Covering ConvNets, Transformers and Everything in Between](https://mlsanity.com/paper/2510.06035/) (2025)
- [Learning to Flow from Generative Pretext Tasks for Neural Architecture Encoding](https://mlsanity.com/paper/2510.18360/) (2025)
- [CoRA-NAS: Coarse Ranking and Anchor-Residual Refinement for Neural Architecture Search](https://mlsanity.com/paper/2609.11884/) (2026)
- [CompressNAS : A Fast and Efficient Technique for Model Compression using Decomposition](https://mlsanity.com/paper/2511.11716/) (2025)
- [GrowTAS: Progressive Expansion from Small to Large Subnets for Efficient ViT Architecture Search](https://mlsanity.com/paper/2512.12296/) (2025)
- [Sequencing to Mitigate Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/2512.16871/) (2025)
- [TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Experts](https://mlsanity.com/paper/2605.07256/) (2026)
- [InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Search](https://mlsanity.com/paper/2606.18676/) (2026)
- [Bi-EZP: LLM-Guided Bilevel Program Evolution for Ensemble Zero-Cost Proxy Discovery](https://mlsanity.com/paper/2608.21927/) (2026)
- [RiPPLE: Cross-Space Performance Prediction from Early Training for Neural Architecture Search](https://mlsanity.com/paper/2609.12418/) (2026)
- [Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone](https://mlsanity.com/paper/2609.23087/) (2026)
- [LESS: Lightweight Evolutionary Supernet Search in Minutes](https://mlsanity.com/paper/2610.01468/) (2026)
- [Bayesian Optimization in Sequence-to-Architecture Latent Space for Zero-Shot NAS](https://mlsanity.com/paper/2610.06167/) (2026)
- [GeoBridge: Evolving Transferable Decoder Topologies for Geometric Vision](https://mlsanity.com/paper/2610.15280/) (2026)
- [FENAS: A Few-Shot Meta-Ensembling Framework for Efficient Neural Architecture Search with Synergized Zero-Cost Proxies](https://mlsanity.com/paper/2610.15307/) (2026)

## Abstract (arXiv)

Training-free network architecture search (NAS) aims to discover high-performing networks with zero-cost proxies, capturing network characteristics related to the final performance. However, network rankings estimated by previous training-free NAS methods have shown weak correlations with the performance. To address this issue, we propose AZ-NAS, a novel approach that leverages the ensemble of various zero-cost proxies to enhance the correlation between a predicted ranking of networks and the ground truth substantially in terms of the performance. To achieve this, we introduce four novel zero-cost proxies that are complementary to each other, analyzing distinct traits of architectures in the views of expressivity, progressivity, trainability, and complexity. The proxy scores can be obtained simultaneously within a single forward and backward pass, making an overall NAS process highly efficient. In order to integrate the rankings predicted by our proxies effectively, we introduce a non-linear ranking aggregation method that highlights the networks highly-ranked consistently across all the proxies. Experimental results conclusively demonstrate the efficacy and efficiency of AZ-NAS, outperforming state-of-the-art methods on standard benchmarks, all while maintaining a reasonable runtime cost.