# Evaluating Efficient Performance Estimators of Neural Architectures

arXiv: https://arxiv.org/abs/2008.03064 · HTML: https://mlsanity.com/paper/2008.03064/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2020.

## Built on by (most cited first, 30 of 55)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients](https://mlsanity.com/paper/2301.11300/) (2023)
- [NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies](https://mlsanity.com/paper/2210.03230/) (2022)
- [AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search](https://mlsanity.com/paper/2403.19232/) (2024)
- [ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies](https://mlsanity.com/paper/2110.10423/) (2021)
- [Joint Neural Architecture and Hyperparameter Search for Correlated Time Series Forecasting](https://mlsanity.com/paper/2211.16126/) (2022)
- [Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities](https://mlsanity.com/paper/2307.01998/) (2023)
- [Analyzing and Mitigating Interference in Neural Architecture Search](https://mlsanity.com/paper/2108.12821/) (2021)
- [DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models](https://mlsanity.com/paper/2305.16943/) (2023)
- [LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models](https://mlsanity.com/paper/2203.02094/) (2022)
- [A Survey on Computationally Efficient Neural Architecture Search](https://mlsanity.com/paper/2206.01520/) (2022)
- ['Eliv'agar: Efficient Quantum Circuit Search for Classification](https://mlsanity.com/paper/2401.09393/) (2024)
- [SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS](https://mlsanity.com/paper/2403.04161/) (2024)
- [Surprisingly Strong Performance Prediction with Neural Graph Features](https://mlsanity.com/paper/2404.16551/) (2024)
- [Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars](https://mlsanity.com/paper/2211.01842/) (2022)
- [RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies](https://mlsanity.com/paper/2301.09850/) (2023)
- [PA&DA: Jointly Sampling PAth and DAta for Consistent NAS](https://mlsanity.com/paper/2302.14772/) (2023)
- [Multi-Predict: Few Shot Predictors For Efficient Neural Architecture Search](https://mlsanity.com/paper/2306.02459/) (2023)
- [CLOSE: Curriculum Learning On the Sharing Extent Towards Better One-shot NAS](https://mlsanity.com/paper/2207.07868/) (2022)
- [ParZC: Parametric Zero-Cost Proxies for Efficient NAS](https://mlsanity.com/paper/2402.02105/) (2024)
- [Transferrable Surrogates in Expressive Neural Architecture Search Spaces](https://mlsanity.com/paper/2504.12971/) (2025)
- [L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformers](https://mlsanity.com/paper/2505.07300/) (2025)
- [Robustifying and Boosting Training-Free Neural Architecture Search](https://mlsanity.com/paper/2403.07591/) (2024)
- [Dynamic Ensemble of Low-fidelity Experts: Mitigating NAS "Cold-Start"](https://mlsanity.com/paper/2302.00932/) (2023)
- [Are Neural Architecture Search Benchmarks Well Designed? A Deeper Look Into Operation Importance](https://mlsanity.com/paper/2303.16938/) (2023)
- [Merino: Entropy-driven Design for Generative Language Models on IoT Devices](https://mlsanity.com/paper/2403.07921/) (2024)
- [Optimizing Time Series Forecasting Architectures: A Hierarchical Neural Architecture Search Approach](https://mlsanity.com/paper/2406.05088/) (2024)
- [Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights](https://mlsanity.com/paper/2502.04975/) (2025)
- [A hierarchical approach for assessing the vulnerability of tree-based classification models to membership inference attack](https://mlsanity.com/paper/2502.09396/) (2025)
- [Guided Evolutionary Neural Architecture Search With Efficient Performance Estimation](https://mlsanity.com/paper/2208.06475/) (2022)

## Abstract (arXiv)

Conducting efficient performance estimations of neural architectures is a major challenge in neural architecture search (NAS). To reduce the architecture training costs in NAS, one-shot estimators (OSEs) amortize the architecture training costs by sharing the parameters of one "supernet" between all architectures. Recently, zero-shot estimators (ZSEs) that involve no training are proposed to further reduce the architecture evaluation cost. Despite the high efficiency of these estimators, the quality of such estimations has not been thoroughly studied. In this paper, we conduct an extensive and organized assessment of OSEs and ZSEs on five NAS benchmarks: NAS-Bench-101/201/301, and NDS ResNet/ResNeXt-A. Specifically, we employ a set of NAS-oriented criteria to study the behavior of OSEs and ZSEs and reveal that they have certain biases and variances. After analyzing how and why the OSE estimations are unsatisfying, we explore how to mitigate the correlation gap of OSEs from several perspectives. Through our analysis, we give out suggestions for future application and development of efficient architecture performance estimators. Furthermore, the analysis framework proposed in our work could be utilized in future research to give a more comprehensive understanding of newly designed architecture performance estimators. All codes are available at https://github.com/walkerning/aw_nas.