Neural Architecture Search: Two Constant Shared Weights Initialisations
Not on the map. Why?
Abstract (arXiv)
In the last decade, zero-cost metrics have gained prominence in neural architecture search (NAS) due to their ability to evaluate architectures without training. These metrics are significantly faster and less computationally expensive than traditional NAS methods and provide insights into neural architectures' internal workings. This paper introduces epsinas, a novel zero-cost NAS metric that assesses architecture potential using two constant shared weight initialisations and the statistics of their outputs. We show that the dispersion of raw outputs, normalised by their average magnitude, strongly correlates with trained accuracy. This effect holds across image classification and language tasks on NAS-Bench-101, NAS-Bench-201, and NAS-Bench-NLP. Our method requires no data labels, operates on a single minibatch, and eliminates the need for gradient computation, making it independent of training hyperparameters, loss metrics, and human annotations. It evaluates a network in a fraction of a GPU second and integrates seamlessly into existing NAS frameworks. The code supporting this study can be found on GitHub at https://github.com/egracheva/epsinas.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 9 Feb 2023.
Builds on
37 of 54 references.
References · 37
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2018DARTS: Differentiable Architecture Searchcited by 1,782
- 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
- 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
- 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
- 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
- 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
- 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
- 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
- 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
25 more
- 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
- 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 258
- 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
- 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
- 2020Neural Architecture Search without Trainingcited by 241
- 2021Zero-Cost Proxies for Lightweight NAScited by 194
- 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 175
- 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
- 2020A Survey on Evolutionary Neural Architecture Searchcited by 142
- 2020DARTS-: Robustly Stepping out of Performance Collapse Without Indicatorscited by 81
- 2019Weight Agnostic Neural Networkscited by 71
- 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
- 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
- 2020Evaluating Efficient Performance Estimators of Neural Architecturescited by 54
- 2021GLiT: Neural Architecture Search for Global and Local Image Transformercited by 53
- 2019AdversarialNAS: Adversarial Neural Architecture Search for GANscited by 38
- 2020Neural Architecture Generator Optimizationcited by 30
- 2017Peephole: Predicting Network Performance Before Trainingcited by 22
- 2021ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxiescited by 20
- 2021GradSign: Model Performance Inference with Theoretical Insightscited by 18
- 2018TAPAS: Train-less Accuracy Predictor for Architecture Searchcited by 11
- 2013Big Neural Networks Waste Capacitycited by 10
- 2021Zero-Cost Operation Scoring in Differentiable Architecture Searchcited by 9
- 2021Multi-objective Optimization by Learning Space Partitionscited by 8
- 2021A Novel Evolutionary Algorithm for Hierarchical Neural Architecture Searchcited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.