# Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks

arXiv: https://arxiv.org/abs/2008.09777 · HTML: https://mlsanity.com/paper/2008.09777/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2020.

## Built on by (most cited first, 30 of 67)

- [CktGNN: Circuit Graph Neural Network for Electronic Design Automation](https://mlsanity.com/paper/2308.16406/) (2023)
- [NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies](https://mlsanity.com/paper/2210.03230/) (2022)
- [AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks](https://mlsanity.com/paper/2306.08107/) (2023)
- [OmniPred: Language Models as Universal Regressors](https://mlsanity.com/paper/2402.14547/) (2024)
- [NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search](https://mlsanity.com/paper/2206.09166/) (2022)
- [Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment](https://mlsanity.com/paper/2208.04321/) (2022)
- [DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models](https://mlsanity.com/paper/2305.16943/) (2023)
- [LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models](https://mlsanity.com/paper/2203.02094/) (2022)
- [Offline Multi-Objective Optimization](https://mlsanity.com/paper/2406.03722/) (2024)
- [EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search](https://mlsanity.com/paper/2210.06015/) (2022)
- [Surprisingly Strong Performance Prediction with Neural Graph Features](https://mlsanity.com/paper/2404.16551/) (2024)
- [FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer](https://mlsanity.com/paper/2403.12821/) (2024)
- [HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models](https://mlsanity.com/paper/2405.10299/) (2024)
- [Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars](https://mlsanity.com/paper/2211.01842/) (2022)
- [Encodings for Prediction-based Neural Architecture Search](https://mlsanity.com/paper/2403.02484/) (2024)
- [AIO-P: Expanding Neural Performance Predictors Beyond Image Classification](https://mlsanity.com/paper/2211.17228/) (2022)
- [SeqNAS: Neural Architecture Search for Event Sequence Classification](https://mlsanity.com/paper/2401.03246/) (2024)
- [Language Model Embeddings Can Be Sufficient for Bayesian Optimization](https://mlsanity.com/paper/2410.10190/) (2024)
- [On Latency Predictors for Neural Architecture Search](https://mlsanity.com/paper/2403.02446/) (2024)
- [LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encoding](https://mlsanity.com/paper/2509.05657/) (2025)
- [Multi-Predict: Few Shot Predictors For Efficient Neural Architecture Search](https://mlsanity.com/paper/2306.02459/) (2023)
- [Robust NAS under adversarial training: benchmark, theory, and beyond](https://mlsanity.com/paper/2403.13134/) (2024)
- [GENNAPE: Towards Generalized Neural Architecture Performance Estimators](https://mlsanity.com/paper/2211.17226/) (2022)
- [CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework](https://mlsanity.com/paper/2212.03965/) (2022)
- [ParZC: Parametric Zero-Cost Proxies for Efficient NAS](https://mlsanity.com/paper/2402.02105/) (2024)
- [Unsupervised Graph Neural Architecture Search with Disentangled Self-supervision](https://mlsanity.com/paper/2403.05064/) (2024)
- [CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework](https://mlsanity.com/paper/2406.01414/) (2024)
- [CAP: A Context-Aware Neural Predictor for NAS](https://mlsanity.com/paper/2406.02056/) (2024)
- [Transferrable Surrogates in Expressive Neural Architecture Search Spaces](https://mlsanity.com/paper/2504.12971/) (2025)
- [L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformers](https://mlsanity.com/paper/2505.07300/) (2025)

## Abstract (arXiv)

The most significant barrier to the advancement of Neural Architecture Search (NAS) is its demand for large computational resources, which hinders scientifically sound empirical evaluations of NAS methods. Tabular NAS benchmarks have alleviated this problem substantially, making it possible to properly evaluate NAS methods in seconds on commodity machines. However, an unintended consequence of tabular NAS benchmarks has been a focus on extremely small architectural search spaces since their construction relies on exhaustive evaluations of the space. This leads to unrealistic results that do not transfer to larger spaces. To overcome this fundamental limitation, we propose a methodology to create cheap NAS surrogate benchmarks for arbitrary search spaces. We exemplify this approach by creating surrogate NAS benchmarks on the existing tabular NAS-Bench-101 and on two widely used NAS search spaces with up to 1021 architectures (1013 times larger than any previous tabular NAS benchmark). We show that surrogate NAS benchmarks can model the true performance of architectures better than tabular benchmarks (at a small fraction of the cost), that they lead to faithful estimates of how well different NAS methods work on the original non-surrogate benchmark, and that they can generate new scientific insight. We open-source all our code and believe that surrogate NAS benchmarks are an indispensable tool to extend scientifically sound work on NAS to large and exciting search spaces.