# Variation Matters: from Mitigating to Embracing Zero-Shot NAS Ranking Function Variation

arXiv: https://arxiv.org/abs/2502.19657 · HTML: https://mlsanity.com/paper/2502.19657/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 27 of 27)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Neural Architecture Search without Training](https://mlsanity.com/paper/2006.04647/) (2020)
- [Zero-Cost Proxies for Lightweight NAS](https://mlsanity.com/paper/2101.08134/) (2021)
- [Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective](https://mlsanity.com/paper/2102.11535/) (2021)
- [TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search](https://mlsanity.com/paper/2105.11871/) (2021)
- [ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients](https://mlsanity.com/paper/2301.11300/) (2023)
- [On the Number of Linear Regions of Convolutional Neural Networks](https://mlsanity.com/paper/2006.00978/) (2020)
- [NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy](https://mlsanity.com/paper/2201.13396/) (2022)
- [KNAS: Green Neural Architecture Search](https://mlsanity.com/paper/2111.13293/) (2021)
- [NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks](https://mlsanity.com/paper/2110.05668/) (2021)
- [BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget](https://mlsanity.com/paper/1906.04113/) (2019)
- [EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search](https://mlsanity.com/paper/2102.08099/) (2021)
- [Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?](https://mlsanity.com/paper/2203.14577/) (2022)
- [EZNAS: Evolving Zero Cost Proxies For Neural Architecture Scoring](https://mlsanity.com/paper/2209.07413/) (2022)
- [LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models](https://mlsanity.com/paper/2203.02094/) (2022)
- [SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS](https://mlsanity.com/paper/2403.04161/) (2024)
- [FreeREA: Training-Free Evolution-based Architecture Search](https://mlsanity.com/paper/2207.05135/) (2022)
- [Training-free Neural Architecture Search for RNNs and Transformers](https://mlsanity.com/paper/2306.00288/) (2023)
- [Entropic Score metric: Decoupling Topology and Size in Training-free NAS](https://mlsanity.com/paper/2310.04179/) (2023)

## Abstract (arXiv)

Neural Architecture Search (NAS) is a powerful automatic alternative to manual design of a neural network. In the zero-shot version, a fast ranking function is used to compare architectures without training them. The outputs of the ranking functions often vary significantly due to different sources of randomness, including the evaluated architecture's weights' initialization or the batch of data used for calculations. A common approach to addressing the variation is to average a ranking function output over several evaluations. We propose taking into account the variation in a different manner, by viewing the ranking function output as a random variable representing a proxy performance metric. During the search process, we strive to construct a stochastic ordering of the performance metrics to determine the best architecture. Our experiments show that the proposed stochastic ordering can effectively boost performance of a search on standard benchmark search spaces.