# NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size

arXiv: https://arxiv.org/abs/2009.00437 · HTML: https://mlsanity.com/paper/2009.00437/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2020.

## Built on by (most cited first, 30 of 98)

- [Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective](https://mlsanity.com/paper/2102.11535/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark](https://mlsanity.com/paper/2103.10584/) (2021)
- [BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2103.12424/) (2021)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients](https://mlsanity.com/paper/2301.11300/) (2023)
- [NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy](https://mlsanity.com/paper/2201.13396/) (2022)
- [Bridging Discrete and Backpropagation: Straight-Through and Beyond](https://mlsanity.com/paper/2304.08612/) (2023)
- [NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search](https://mlsanity.com/paper/2206.09166/) (2022)
- [ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies](https://mlsanity.com/paper/2110.10423/) (2021)
- [Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities](https://mlsanity.com/paper/2307.01998/) (2023)
- [NAS-Bench-x11 and the Power of Learning Curves](https://mlsanity.com/paper/2111.03602/) (2021)
- [PyGlove: Symbolic Programming for Automated Machine Learning](https://mlsanity.com/paper/2101.08809/) (2021)
- [Isometric Propagation Network for Generalized Zero-shot Learning](https://mlsanity.com/paper/2102.02038/) (2021)
- [Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment](https://mlsanity.com/paper/2208.04321/) (2022)
- [Few-Shot Learning with a Strong Teacher](https://mlsanity.com/paper/2107.00197/) (2021)
- [EZNAS: Evolving Zero Cost Proxies For Neural Architecture Scoring](https://mlsanity.com/paper/2209.07413/) (2022)
- [Accelerating Neural Architecture Search via Proxy Data](https://mlsanity.com/paper/2106.04784/) (2021)
- [Generalization Properties of NAS under Activation and Skip Connection Search](https://mlsanity.com/paper/2209.07238/) (2022)
- [Offline Multi-Objective Optimization](https://mlsanity.com/paper/2406.03722/) (2024)
- [OptunaHub: A Platform for Black-Box Optimization](https://mlsanity.com/paper/2510.02798/) (2025)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [Order-Preserving GFlowNets](https://mlsanity.com/paper/2310.00386/) (2023)
- [Prior-Guided One-shot Neural Architecture Search](https://mlsanity.com/paper/2206.13329/) (2022)
- [A Benchmark Study on Calibration](https://mlsanity.com/paper/2308.11838/) (2023)
- [FreeREA: Training-Free Evolution-based Architecture Search](https://mlsanity.com/paper/2207.05135/) (2022)
- [Efficient Lifelong Model Evaluation in an Era of Rapid Progress](https://mlsanity.com/paper/2402.19472/) (2024)
- [NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters](https://mlsanity.com/paper/2110.10165/) (2021)
- [Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars](https://mlsanity.com/paper/2211.01842/) (2022)

## Abstract (arXiv)

Neural architecture search (NAS) has attracted a lot of attention and has been illustrated to bring tangible benefits in a large number of applications in the past few years. Architecture topology and architecture size have been regarded as two of the most important aspects for the performance of deep learning models and the community has spawned lots of searching algorithms for both aspects of the neural architectures. However, the performance gain from these searching algorithms is achieved under different search spaces and training setups. This makes the overall performance of the algorithms to some extent incomparable and the improvement from a sub-module of the searching model unclear. In this paper, we propose NATS-Bench, a unified benchmark on searching for both topology and size, for (almost) any up-to-date NAS algorithm. NATS-Bench includes the search space of 15,625 neural cell candidates for architecture topology and 32,768 for architecture size on three datasets. We analyze the validity of our benchmark in terms of various criteria and performance comparison of all candidates in the search space. We also show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms on it. All logs and diagnostic information trained using the same setup for each candidate are provided. This facilitates a much larger community of researchers to focus on developing better NAS algorithms in a more comparable and computationally cost friendly environment. All codes are publicly available at: https://xuanyidong.com/assets/projects/NATS-Bench.