mlsanityData to 7 Oct 2026

AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search

arXiv 2403.19232 ↗First posted 28 Mar 2024cs.CV cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Training-free network architecture search (NAS) aims to discover high-performing networks with zero-cost proxies, capturing network characteristics related to the final performance. However, network rankings estimated by previous training-free NAS methods have shown weak correlations with the performance. To address this issue, we propose AZ-NAS, a novel approach that leverages the ensemble of various zero-cost proxies to enhance the correlation between a predicted ranking of networks and the ground truth substantially in terms of the performance. To achieve this, we introduce four novel zero-cost proxies that are complementary to each other, analyzing distinct traits of architectures in the views of expressivity, progressivity, trainability, and complexity. The proxy scores can be obtained simultaneously within a single forward and backward pass, making an overall NAS process highly efficient. In order to integrate the rankings predicted by our proxies effectively, we introduce a non-linear ranking aggregation method that highlights the networks highly-ranked consistently across all the proxies. Experimental results conclusively demonstrate the efficacy and efficiency of AZ-NAS, outperforming state-of-the-art methods on standard benchmarks, all while maintaining a reasonable runtime cost.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 61 of its 80 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

61 of 80 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  12. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
38 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  4. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  5. 2017Random Erasing Data Augmentationcited by 1,235
  6. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  10. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  11. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  12. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  13. 2017Progressive Neural Architecture Searchcited by 483
  14. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  15. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  16. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  17. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  18. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  19. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  20. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  21. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  22. 2020Neural Architecture Search without Trainingcited by 241
  23. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  24. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  25. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  26. 2019Complexity of Linear Regions in Deep Networkscited by 166
  27. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  28. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  29. 2015Reducing Overfitting in Deep Networks by Decorrelating Representationscited by 94
  30. 2021Network Quantization with Element-wise Gradient Scalingcited by 82
  31. 2019CARS: Continuous Evolution for Efficient Neural Architecture Searchcited by 79
  32. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
  33. 2019On Network Design Spaces for Visual Recognitioncited by 68
  34. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  35. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  36. 2020Evaluating Efficient Performance Estimators of Neural Architecturescited by 54
  37. 2020Few-shot Neural Architecture Searchcited by 50
  38. 2020On the Number of Linear Regions of Convolutional Neural Networkscited by 49

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 9.

25 papers.

Sort:

By area:

  1. 2026Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedbackcited by 8
  2. 2025L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformerscited by 5
  3. 2025Subnet-Aware Dynamic Supernet Training for Neural Architecture Searchcited by 4
  4. 2024Towards certifiable AI in aviation: landscape, challenges, and opportunitiescited by 2
  5. 2024Zero-Shot NAS via the Suppression of Local Entropy Decreasecited by 2
  6. 2025Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weightscited by 2
  7. 2025CARL: Causality-guided Architecture Representation Learning for an Interpretable Performance Predictorcited by 2
  8. 2026Bilevel Optimization for Neural Architecture Searchcited by 2
  9. 2024GreenMachine: Automatic Design of Zero-Cost Proxies for Energy-Efficient NAScited by 1
  10. 2025Zero-Shot Neural Architecture Search with Weighted Response Correlationcited by 1
  11. 2025QuProFS: An Evolutionary Training-free Approach to Efficient Quantum Feature Map Searchcited by 1
  12. 2025Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvaturecited by 1
13 more
  1. 2025LLM-NAS: LLM-driven Hardware-Aware Neural Architecture Searchcited by 1
  2. 2025Learning to Flow from Generative Pretext Tasks for Neural Architecture Encodingcited by 1
  3. 2026CoRA-NAS: Coarse Ranking and Anchor-Residual Refinement for Neural Architecture Searchcited by 1
  4. 2025Universal Neural Architecture Space: Covering ConvNets, Transformers and Everything in Betweencited by 0
  5. 2025CompressNAS : A Fast and Efficient Technique for Model Compression using Decompositioncited by 0
  6. 2025GrowTAS: Progressive Expansion from Small to Large Subnets for Efficient ViT Architecture Searchcited by 0
  7. 2025Sequencing to Mitigate Catastrophic Forgetting in Continual Learningcited by 0
  8. 2026TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Expertscited by 0
  9. 2026InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Searchcited by 0
  10. 2026Bi-EZP: LLM-Guided Bilevel Program Evolution for Ensemble Zero-Cost Proxy Discoverycited by 0
  11. 2026RiPPLE: Cross-Space Performance Prediction from Early Training for Neural Architecture Searchcited by 0
  12. 2026Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alonecited by 0
  13. 2026LESS: Lightweight Evolutionary Supernet Search in Minutescited by 0