# ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients

arXiv: https://arxiv.org/abs/2301.11300 · HTML: https://mlsanity.com/paper/2301.11300/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 95 of its 116 references that are in the map.
- First posted on arXiv 26 Jan 2023.

## Builds on (most cited first, 30 of 95)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks](https://mlsanity.com/paper/1905.07953/) (2019)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)

## Built on by (most cited first, 30 of 58)

- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search](https://mlsanity.com/paper/2403.19232/) (2024)
- [Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities](https://mlsanity.com/paper/2307.01998/) (2023)
- [SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS](https://mlsanity.com/paper/2403.04161/) (2024)
- [No More Adam: Learning Rate Scaling at Initialization is All You Need](https://mlsanity.com/paper/2412.11768/) (2024)
- [LPZero: Language Model Zero-cost Proxy Search from Zero](https://mlsanity.com/paper/2410.04808/) (2024)
- [OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators](https://mlsanity.com/paper/2312.09411/) (2023)
- [YOLOBench: Benchmarking Efficient Object Detectors on Embedded Systems](https://mlsanity.com/paper/2307.13901/) (2023)
- [Efficient Training of Large Vision Models via Advanced Automated Progressive Learning](https://mlsanity.com/paper/2410.00350/) (2024)
- [L-SWAG: Layer-Sample Wise Activation with Gradients information for Zero-Shot NAS on Vision Transformers](https://mlsanity.com/paper/2505.07300/) (2025)
- [MONAS: Efficient Zero-Shot Neural Architecture Search for MCUs](https://mlsanity.com/paper/2408.15034/) (2024)
- [Subnet-Aware Dynamic Supernet Training for Neural Architecture Search](https://mlsanity.com/paper/2503.10740/) (2025)
- [RBFleX-NAS: Training-Free Neural Architecture Search Using Radial Basis Function Kernel and Hyperparameter Detection](https://mlsanity.com/paper/2503.22733/) (2025)
- [DCLP: Neural Architecture Predictor with Curriculum Contrastive Learning](https://mlsanity.com/paper/2302.13020/) (2023)
- [HyTAS: A Hyperspectral Image Transformer Architecture Search Benchmark and Analysis](https://mlsanity.com/paper/2407.16269/) (2024)
- [Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights](https://mlsanity.com/paper/2502.04975/) (2025)
- [pTNAS: Progressive Neural Architecture Search for Tabular Data](https://mlsanity.com/paper/2403.10318/) (2024)
- [TG-NAS: Generalizable Zero-Cost Proxies with Operator Description Embedding and Graph Learning for Efficient Neural Architecture Search](https://mlsanity.com/paper/2404.00271/) (2024)
- [Differentiable Model Scaling using Differentiable Topk](https://mlsanity.com/paper/2405.07194/) (2024)
- [Zero-Shot NAS via the Suppression of Local Entropy Decrease](https://mlsanity.com/paper/2411.06236/) (2024)
- [Neural Architecture Search: Two Constant Shared Weights Initialisations](https://mlsanity.com/paper/2302.04406/) (2023)
- [Learning Activation Functions for Sparse Neural Networks](https://mlsanity.com/paper/2305.10964/) (2023)
- [ZiCo-BC: A Bias Corrected Zero-Shot NAS for Vision Tasks](https://mlsanity.com/paper/2309.14666/) (2023)
- [SiGeo: Sub-One-Shot NAS via Information Theory and Geometry of Loss Landscape](https://mlsanity.com/paper/2311.13169/) (2023)
- [Graph is all you need? Lightweight data-agnostic neural architecture search without training](https://mlsanity.com/paper/2405.01306/) (2024)
- [Federated Learning with Flexible Architectures](https://mlsanity.com/paper/2406.09877/) (2024)
- [NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance](https://mlsanity.com/paper/2408.08776/) (2024)
- [GreenMachine: Automatic Design of Zero-Cost Proxies for Energy-Efficient NAS](https://mlsanity.com/paper/2411.15290/) (2024)
- [W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models](https://mlsanity.com/paper/2504.15983/) (2025)
- [GreenFactory: Ensembling Zero-Cost Proxies to Estimate Performance of Neural Networks](https://mlsanity.com/paper/2505.09344/) (2025)

## Abstract (arXiv)

Neural Architecture Search (NAS) is widely used to automatically obtain the neural network with the best performance among a large number of candidate architectures. To reduce the search time, zero-shot NAS aims at designing training-free proxies that can predict the test performance of a given architecture. However, as shown recently, none of the zero-shot proxies proposed to date can actually work consistently better than a naive proxy, namely, the number of network parameters (#Params). To improve this state of affairs, as the main theoretical contribution, we first reveal how some specific gradient properties across different samples impact the convergence rate and generalization capacity of neural networks. Based on this theoretical analysis, we propose a new zero-shot proxy, ZiCo, the first proxy that works consistently better than #Params. We demonstrate that ZiCo works better than State-Of-The-Art (SOTA) proxies on several popular NAS-Benchmarks (NASBench101, NATSBench-SSS/TSS, TransNASBench-101) for multiple applications (e.g., image classification/reconstruction and pixel-level prediction). Finally, we demonstrate that the optimal architectures found via ZiCo are as competitive as the ones found by one-shot and multi-shot NAS methods, but with much less search time. For example, ZiCo-based NAS can find optimal architectures with 78.1%, 79.4%, and 80.4% test accuracy under inference budgets of 450M, 600M, and 1000M FLOPs, respectively, on ImageNet within 0.4 GPU days. Our code is available at https://github.com/SLDGroup/ZiCo.