mlsanityData to 14 Oct 2026

Variation Matters: from Mitigating to Embracing Zero-Shot NAS Ranking Function Variation

arXiv 2502.19657 ↗First posted 27 Feb 2025cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural Architecture Search (NAS) is a powerful automatic alternative to manual design of a neural network. In the zero-shot version, a fast ranking function is used to compare architectures without training them. The outputs of the ranking functions often vary significantly due to different sources of randomness, including the evaluated architecture's weights' initialization or the batch of data used for calculations. A common approach to addressing the variation is to average a ranking function output over several evaluations. We propose taking into account the variation in a different manner, by viewing the ranking function output as a random variable representing a proxy performance metric. During the search process, we strive to construct a stochastic ordering of the performance metrics to determine the best architecture. Our experiments show that the proposed stochastic ordering can effectively boost performance of a search on standard benchmark search spaces.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

27 of 36 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  5. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  6. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  7. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  8. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  9. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
  10. 2020Neural Architecture Search without Trainingcited by 244
  11. 2021Zero-Cost Proxies for Lightweight NAScited by 199
  12. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
15 more
  1. 2021TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Searchcited by 58
  2. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 58
  3. 2020On the Number of Linear Regions of Convolutional Neural Networkscited by 49
  4. 2022NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easycited by 36
  5. 2021KNAS: Green Neural Architecture Searchcited by 33
  6. 2021NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Taskscited by 32
  7. 2019BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budgetcited by 26
  8. 2021EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Searchcited by 24
  9. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  10. 2022EZNAS: Evolving Zero Cost Proxies For Neural Architecture Scoringcited by 15
  11. 2022LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Modelscited by 14
  12. 2024SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAScited by 11
  13. 2022FreeREA: Training-Free Evolution-based Architecture Searchcited by 10
  14. 2023Training-free Neural Architecture Search for RNNs and Transformerscited by 9
  15. 2023Entropic Score metric: Decoupling Topology and Size in Training-free NAScited by 2

Built on by

Not cited yet.