Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights
Not on the map. Why?
Abstract (arXiv)
Deep learning has revolutionized computer vision, but it achieved its tremendous success using deep network architectures which are mostly hand-crafted and therefore likely suboptimal. Neural Architecture Search (NAS) aims to bridge this gap by following a well-defined optimization paradigm which systematically looks for the best architecture, given objective criterion such as maximal classification accuracy. The main limitation of NAS is however its astronomical computational cost, as it typically requires training each candidate network architecture from scratch. In this paper, we aim to alleviate this limitation by proposing a novel training-free proxy for image classification accuracy based on Fisher Information. The proposed proxy has a strong theoretical background in statistics and it allows estimating expected image classification accuracy of a given deep network without training the network, thus significantly reducing computational cost of standard NAS algorithms. Our training-free proxy achieves state-of-the-art results on three public datasets and in two search spaces, both when evaluated using previously proposed metrics, as well as using a new metric that we propose which we demonstrate is more informative for practical NAS applications. The source code is publicly available at http://www.github.com/ondratybl/VKDNW
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 7 Feb 2025.
Builds on
34 of 50 references.
References · 34
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2018DARTS: Differentiable Architecture Searchcited by 1,798
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
- 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
- 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
- 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
- 2014New insights and perspectives on the natural gradient methodcited by 443
- 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
22 more
- 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
- 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 344
- 2020Neural Architecture Search without Trainingcited by 244
- 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
- 2021Zero-Cost Proxies for Lightweight NAScited by 199
- 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
- 2019Complexity of Linear Regions in Deep Networkscited by 167
- 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 91
- 2019CARS: Continuous Evolution for Efficient Neural Architecture Searchcited by 79
- 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 73
- 2022Fast Finite Width Neural Tangent Kernelcited by 62
- 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 58
- 2020Evaluating Efficient Performance Estimators of Neural Architecturescited by 55
- 2020On the Number of Linear Regions of Convolutional Neural Networkscited by 49
- 2021Neural Architecture Search with Random Labelscited by 33
- 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
- 2020Distilling Optimal Neural Networks: Rapid Search in Diverse Spacescited by 32
- 2024AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Searchcited by 28
- 2021BN-NAS: Neural Architecture Search with Batch Normalizationcited by 21
- 2022Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Networkcited by 21
- 2021GradSign: Model Performance Inference with Theoretical Insightscited by 19
- 2024Surprisingly Strong Performance Prediction with Neural Graph Featurescited by 11
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.