# How Powerful are Performance Predictors in Neural Architecture Search?

arXiv: https://arxiv.org/abs/2104.01177 · HTML: https://mlsanity.com/paper/2104.01177/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 88 references that are in the map.
- First posted on arXiv 2 Apr 2021.

## Builds on (most cited first, 30 of 50)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Searching for A Robust Neural Architecture in Four GPU Hours](https://mlsanity.com/paper/1910.04465/) (2019)
- [Random Search and Reproducibility for Neural Architecture Search](https://mlsanity.com/paper/1902.07638/) (2019)
- [Neural Architecture Search without Training](https://mlsanity.com/paper/2006.04647/) (2020)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Zero-Cost Proxies for Lightweight NAS](https://mlsanity.com/paper/2101.08134/) (2021)
- [Neural Architecture Search with Bayesian Optimisation and Optimal Transport](https://mlsanity.com/paper/1802.07191/) (2018)
- [BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search](https://mlsanity.com/paper/1910.11858/) (2019)
- [Understanding and Robustifying Differentiable Architecture Search](https://mlsanity.com/paper/1909.09656/) (2019)
- [Neural Architecture Optimization](https://mlsanity.com/paper/1808.07233/) (2018)
- [Evaluating the Search Phase of Neural Architecture Search](https://mlsanity.com/paper/1902.08142/) (2019)
- [Neural Predictor for Neural Architecture Search](https://mlsanity.com/paper/1912.00848/) (2019)
- [DARTS+: Improved Differentiable Architecture Search with Early Stopping](https://mlsanity.com/paper/1909.06035/) (2019)
- [Faster gaze prediction with dense networks and Fisher pruning](https://mlsanity.com/paper/1801.05787/) (2018)

## Built on by (most cited first, 30 of 90)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2301.12132/) (2023)
- [NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy](https://mlsanity.com/paper/2201.13396/) (2022)
- [NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies](https://mlsanity.com/paper/2210.03230/) (2022)
- [AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks](https://mlsanity.com/paper/2306.08107/) (2023)
- [PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning](https://mlsanity.com/paper/2306.12370/) (2023)
- [ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies](https://mlsanity.com/paper/2110.10423/) (2021)
- [Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities](https://mlsanity.com/paper/2307.01998/) (2023)
- [NAS-Bench-x11 and the Power of Learning Curves](https://mlsanity.com/paper/2111.03602/) (2021)
- [Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?](https://mlsanity.com/paper/2203.14577/) (2022)
- [DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models](https://mlsanity.com/paper/2305.16943/) (2023)
- [Learning Where To Look -- Generative NAS is Surprisingly Efficient](https://mlsanity.com/paper/2203.08734/) (2022)
- [AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models](https://mlsanity.com/paper/2509.12019/) (2025)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?](https://mlsanity.com/paper/2205.11656/) (2022)
- [A Survey on Computationally Efficient Neural Architecture Search](https://mlsanity.com/paper/2206.01520/) (2022)
- [Surprisingly Strong Performance Prediction with Neural Graph Features](https://mlsanity.com/paper/2404.16551/) (2024)
- [Regression Language Models for Code](https://mlsanity.com/paper/2509.26476/) (2025)
- [FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer](https://mlsanity.com/paper/2403.12821/) (2024)
- [HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models](https://mlsanity.com/paper/2405.10299/) (2024)
- [Encodings for Prediction-based Neural Architecture Search](https://mlsanity.com/paper/2403.02484/) (2024)
- [Speeding up NAS with Adaptive Subset Selection](https://mlsanity.com/paper/2211.01454/) (2022)
- [Is One Epoch All You Need For Multi-Fidelity Hyperparameter Optimization?](https://mlsanity.com/paper/2307.15422/) (2023)
- [Efficient Activation Function Optimization through Surrogate Modeling](https://mlsanity.com/paper/2301.05785/) (2023)
- [Building Optimal Neural Architectures using Interpretable Knowledge](https://mlsanity.com/paper/2403.13293/) (2024)
- [Structural Pruning of Pre-trained Language Models via Neural Architecture Search](https://mlsanity.com/paper/2405.02267/) (2024)
- [NeurDB: An AI-powered Autonomous Data System](https://mlsanity.com/paper/2405.03924/) (2024)
- [Improving Hyperparameter Optimization with Checkpointed Model Weights](https://mlsanity.com/paper/2406.18630/) (2024)
- [On Accelerating Edge AI: Optimizing Resource-Constrained Environments](https://mlsanity.com/paper/2501.15014/) (2025)
- [Transferrable Surrogates in Expressive Neural Architecture Search Spaces](https://mlsanity.com/paper/2504.12971/) (2025)

## Abstract (arXiv)

Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks. To reduce this extreme computational cost, dozens of techniques have since been proposed to predict the final performance of neural architectures. Despite the success of such performance prediction methods, it is not well-understood how different families of techniques compare to one another, due to the lack of an agreed-upon evaluation metric and optimization for different constraints on the initialization time and query time. In this work, we give the first large-scale study of performance predictors by analyzing 31 techniques ranging from learning curve extrapolation, to weight-sharing, to supervised learning, to "zero-cost" proxies. We test a number of correlation- and rank-based performance measures in a variety of settings, as well as the ability of each technique to speed up predictor-based NAS frameworks. Our results act as recommendations for the best predictors to use in different settings, and we show that certain families of predictors can be combined to achieve even better predictive power, opening up promising research directions. Our code, featuring a library of 31 performance predictors, is available at https://github.com/automl/naslib.