mlsanityData to 7 Oct 2026

NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy

arXiv 2201.13396 ↗First posted 31 Jan 2022cs.LG cs.AI stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS). Although they have been widely adopted and used to tune real-world NAS algorithms, these benchmarks are limited to small search spaces and focus solely on image classification. Recently, several new NAS benchmarks have been introduced that cover significantly larger search spaces over a wide range of tasks, including object detection, speech recognition, and natural language processing. However, substantial differences among these NAS benchmarks have so far prevented their widespread adoption, limiting researchers to using just a few benchmarks. In this work, we present an in-depth analysis of popular NAS algorithms and performance prediction methods across 25 different combinations of search spaces and datasets, finding that many conclusions drawn from a few NAS benchmarks do not generalize to other benchmarks. To help remedy this problem, we introduce NAS-Bench-Suite, a comprehensive and extensible collection of NAS benchmarks, accessible through a unified interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research. Our code is available at https://github.com/automl/naslib.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 46 of its 66 references that are in the map. First posted on arXiv 31 Jan 2022.

Builds on

46 of 66 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  3. 2018How Powerful are Graph Neural Networks?cited by 4,355
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  6. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  8. 2018A Tutorial on Bayesian Optimizationcited by 734
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  11. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  12. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
34 more
  1. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  2. 2019Tackling Climate Change with Machine Learningcited by 264
  3. 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 258
  4. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  5. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  6. 2019The Evolved Transformercited by 180
  7. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  8. 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 175
  9. 2018Neural Architecture Optimizationcited by 164
  10. 2019Evaluating the Search Phase of Neural Architecture Searchcited by 120
  11. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  12. 2020NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Sizecited by 98
  13. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 88
  14. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  15. 2020DrNAS: Dirichlet Neural Architecture Searchcited by 73
  16. 2019NAS evaluation is frustratingly hardcited by 69
  17. 2020NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Searchcited by 64
  18. 2020NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processingcited by 60
  19. 2021TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Searchcited by 57
  20. 2019Best Practices for Scientific Research on Neural Architecture Searchcited by 53
  21. 2019Densely Connected Search Space for More Flexible Neural Architecture Searchcited by 52
  22. 2020Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gapcited by 50
  23. 2020Semi-Supervised Neural Architecture Searchcited by 45
  24. 2020A Study on Encodings for Neural Architecture Searchcited by 37
  25. 2021Prioritized Architecture Sampling with Monto-Carlo Tree Searchcited by 32
  26. 2020Neural Architecture Generator Optimizationcited by 30
  27. 2017DeepArchitect: Automatically Designing and Training Deep Architecturescited by 28
  28. 2020Exploring the Loss Landscape in Neural Architecture Searchcited by 20
  29. 2021NAS-Bench-x11 and the Power of Learning Curvescited by 20
  30. 2021PyGlove: Symbolic Programming for Automated Machine Learningcited by 19
  31. 2020How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAScited by 17
  32. 2020Local Search is a Remarkably Strong Baseline for Neural Architecture Searchcited by 15
  33. 2021It's Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Processcited by 11
  34. 2021Learning Versatile Neural Architectures by Propagating Network Codescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 11; 2024: 9; 2025: 6; 2026: 2.

36 papers.

Sort:

  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  2. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  3. 2022NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxiescited by 36
  4. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  5. 2023Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunitiescited by 21
  6. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  7. 2022Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessmentcited by 17
  8. 2022Learning Where To Look -- Generative NAS is Surprisingly Efficientcited by 15
  9. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  10. 2023Training-free Neural Architecture Search for RNNs and Transformerscited by 9
  11. 2024Model Parallelism on Distributed Infrastructure: A Literature Review from Theory to LLM Case-Studiescited by 7
  12. 2022TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasetscited by 6
24 more
  1. 2024Building Optimal Neural Architectures using Interpretable Knowledgecited by 5
  2. 2024Insights from the Use of Previously Unseen Neural Architecture Search Datasetscited by 5
  3. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  4. 2022Exploring the Intersection between Neural Architecture Search and Continual Learningcited by 4
  5. 2022AutoML for Climate Change: A Call to Actioncited by 3
  6. 2023Are Neural Architecture Search Benchmarks Well Designed? A Deeper Look Into Operation Importancecited by 3
  7. 2023Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gymcited by 3
  8. 2024Optimizing Time Series Forecasting Architectures: A Hierarchical Neural Architecture Search Approachcited by 3
  9. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
  10. 2022Shortest Edit Path Crossover: A Theory-driven Solution to the Permutation Problem in Evolutionary Neural Architecture Searchcited by 2
  11. 2023Python Wrapper for Simulating Multi-Fidelity Optimization on HPO Benchmarks without Any Waitcited by 2
  12. 2023Rethink DARTS Search Space and Renovate a New Benchmarkcited by 2
  13. 2024Fast Benchmarking of Asynchronous Multi-Fidelity Optimization on Zero-Cost Benchmarkscited by 2
  14. 2025MARCO: Hardware-Aware Neural Architecture Search for Edge Devices with Multi-Agent Reinforcement Learning and Conformal Prediction Filteringcited by 2
  15. 2022The BUTTER Zone: An Empirical Study of Training Dynamics in Fully Connected Neural Networkscited by 1
  16. 2023ZiCo-BC: A Bias Corrected Zero-Shot NAS for Vision Taskscited by 1
  17. 2025Omnipresent Yet Overlooked: Heat Kernels in Combinatorial Bayesian Optimizationcited by 1
  18. 2023Machine learning's own Industrial Revolutioncited by 0
  19. 2024FR-NAS: Forward-and-Reverse Graph Predictor for Efficient Neural Architecture Searchcited by 0
  20. 2025Variation Matters: from Mitigating to Embracing Zero-Shot NAS Ranking Function Variationcited by 0
  21. 2025confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methodscited by 0
  22. 2025Best Practices For Empirical Meta-Algorithmic Research: Guidelines from the COSEAL Research Networkcited by 0
  23. 2026LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systemscited by 0
  24. 2026Energy Accuracy Is Not Enough: A Structure-Aware Benchmark and Evaluation Protocol for Quantum Architecture Searchcited by 0