mlsanityData to 7 Oct 2026

How Powerful are Performance Predictors in Neural Architecture Search?

arXiv 2104.01177 ↗First posted 2 Apr 2021cs.LG cs.NE stat.MLcited by 88

Not on the map. Why?

Abstract (arXiv)

Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks. To reduce this extreme computational cost, dozens of techniques have since been proposed to predict the final performance of neural architectures. Despite the success of such performance prediction methods, it is not well-understood how different families of techniques compare to one another, due to the lack of an agreed-upon evaluation metric and optimization for different constraints on the initialization time and query time. In this work, we give the first large-scale study of performance predictors by analyzing 31 techniques ranging from learning curve extrapolation, to weight-sharing, to supervised learning, to "zero-cost" proxies. We test a number of correlation- and rank-based performance measures in a variety of settings, as well as the ability of each technique to speed up predictor-based NAS frameworks. Our results act as recommendations for the best predictors to use in different settings, and we show that certain families of predictors can be combined to achieve even better predictive power, opening up promising research directions. Our code, featuring a library of 31 performance predictors, is available at https://github.com/automl/naslib.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 50 of its 88 references that are in the map. First posted on arXiv 2 Apr 2021.

Builds on

50 of 88 references.

References · 50

  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  2. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  4. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  5. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  6. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  8. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  9. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  10. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  11. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  12. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
38 more
  1. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  2. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  3. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  4. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  5. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  6. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  7. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  8. 2020Neural Architecture Search without Trainingcited by 241
  9. 2018A System for Massively Parallel Hyperparameter Tuningcited by 201
  10. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  11. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
  12. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  13. 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 175
  14. 2018Neural Architecture Optimizationcited by 164
  15. 2019Evaluating the Search Phase of Neural Architecture Searchcited by 120
  16. 2019Neural Predictor for Neural Architecture Searchcited by 116
  17. 2019DARTS+: Improved Differentiable Architecture Search with Early Stoppingcited by 105
  18. 2019NGBoost: Natural Gradient Boosting for Probabilistic Predictioncited by 94
  19. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
  20. 2019A Survey on Neural Architecture Searchcited by 71
  21. 2019NAS evaluation is frustratingly hardcited by 69
  22. 2016Understanding Probabilistic Sparse Gaussian Process Approximationscited by 68
  23. 2020NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Searchcited by 64
  24. 2020EcoNAS: Finding Proxies for Economical Neural Architecture Searchcited by 61
  25. 2020NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processingcited by 60
  26. 2019Best Practices for Scientific Research on Neural Architecture Searchcited by 53
  27. 2020A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAScited by 53
  28. 2020Semi-Supervised Neural Architecture Searchcited by 45
  29. 2020Geometry-Aware Gradient Algorithms for Neural Architecture Searchcited by 40
  30. 2020A Study on Encodings for Neural Architecture Searchcited by 37
  31. 2019AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Searchcited by 33
  32. 2020Deeper Insights into Weight Sharing in Neural Architecture Searchcited by 24
  33. 2019Deep Neural Architecture Search with Deep Graph Bayesian Optimizationcited by 17
  34. 2020How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAScited by 17
  35. 2014Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spacescited by 16
  36. 2018Evolutionary-Neural Hybrid Agents for Architecture Searchcited by 11
  37. 2020A Novel Training Protocol for Performance Predictors of Evolutionary Neural Architecture Search Algorithmscited by 9
  38. 2020Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 20; 2023: 16; 2024: 18; 2025: 15; 2026: 10.

88 papers.

Sort:

By area:

  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  2. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  3. 2022NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easycited by 36
  4. 2022NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxiescited by 36
  5. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  6. 2023PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learningcited by 31
  7. 2023Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunitiescited by 21
  8. 2021ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxiescited by 20
  9. 2021NAS-Bench-x11 and the Power of Learning Curvescited by 20
  10. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  11. 2022Learning Where To Look -- Generative NAS is Surprisingly Efficientcited by 15
  12. 2023DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Modelscited by 15
38 more
  1. 2025AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Modelscited by 15
  2. 2022FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid?cited by 12
  3. 2022A Survey on Computationally Efficient Neural Architecture Searchcited by 12
  4. 2022Neural Architecture Search for Dense Prediction Tasks in Computer Visioncited by 11
  5. 2024Surprisingly Strong Performance Prediction with Neural Graph Featurescited by 11
  6. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  7. 2025Regression Language Models for Codecited by 10
  8. 2024Encodings for Prediction-based Neural Architecture Searchcited by 9
  9. 2024FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformercited by 9
  10. 2022Speeding up NAS with Adaptive Subset Selectioncited by 6
  11. 2023Is One Epoch All You Need For Multi-Fidelity Hyperparameter Optimization?cited by 6
  12. 2023Efficient Activation Function Optimization through Surrogate Modelingcited by 5
  13. 2024Building Optimal Neural Architectures using Interpretable Knowledgecited by 5
  14. 2024Structural Pruning of Pre-trained Language Models via Neural Architecture Searchcited by 5
  15. 2024NeurDB: An AI-powered Autonomous Data Systemcited by 5
  16. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  17. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  18. 2025Transferrable Surrogates in Expressive Neural Architecture Search Spacescited by 5
  19. 2021Differentiable Architecture Search for Reinforcement Learningcited by 4
  20. 2022Towards Less Constrained Macro-Neural Architecture Searchcited by 4
  21. 2022Exploring the Intersection between Neural Architecture Search and Continual Learningcited by 4
  22. 2022Pareto-aware Neural Architecture Generation for Diverse Computational Budgetscited by 4
  23. 2023A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Modelscited by 4
  24. 2024Robustifying and Boosting Training-Free Neural Architecture Searchcited by 4
  25. 2025Modular Machine Learning: An Indispensable Path towards New-Generation Large Language Modelscited by 4
  26. 2021Mutation is all you needcited by 3
  27. 2021Guided Evolution for Neural Architecture Searchcited by 3
  28. 2022Tackling Neural Architecture Search With Quality Diversity Optimizationcited by 3
  29. 2023Dynamic Ensemble of Low-fidelity Experts: Mitigating NAS "Cold-Start"cited by 3
  30. 2023Are Neural Architecture Search Benchmarks Well Designed? A Deeper Look Into Operation Importancecited by 3
  31. 2025Multi-View Encoders for Performance Prediction in LLM-Based Agentic Workflowscited by 3
  32. 2021BINAS: Bilinear Interpretable Neural Architecture Searchcited by 2
  33. 2021Training BatchNorm Only in Neural Architecture Search and Beyondcited by 2
  34. 2022Guided Evolutionary Neural Architecture Search With Efficient Performance Estimationcited by 2
  35. 2023HyT-NAS: Hybrid Transformers Neural Architecture Search for Edge Devicescited by 2
  36. 2023Pruning at Initialization -- A Sketching Perspectivecited by 2
  37. 2023LISSNAS: Locality-based Iterative Search Space Shrinkage for Neural Architecture Searchcited by 2
  38. 2023How Much Is Hidden in the NAS Benchmarks? Few-Shot Adaptation of a NAS Predictorcited by 2

50 most cited of 88 All 88 (JSON)