# KNAS: Green Neural Architecture Search

arXiv: https://arxiv.org/abs/2111.13293 · HTML: https://mlsanity.com/paper/2111.13293/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 63 references that are in the map.
- First posted on arXiv 26 Nov 2021.

## Builds on (most cited first, 30 of 53)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks](https://mlsanity.com/paper/1901.08584/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers](https://mlsanity.com/paper/1811.04918/) (2018)
- [On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization](https://mlsanity.com/paper/1802.06509/) (2018)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)

## Built on by (most cited first, 30 of 33)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients](https://mlsanity.com/paper/2301.11300/) (2023)
- [Neural Architecture Search for Spiking Neural Networks](https://mlsanity.com/paper/2201.10355/) (2022)
- [Quantum Architecture Search: A Survey](https://mlsanity.com/paper/2406.06210/) (2024)
- [DisWOT: Student Architecture Search for Distillation WithOut Training](https://mlsanity.com/paper/2303.15678/) (2023)
- [LEMON: Lossless model expansion](https://mlsanity.com/paper/2310.07999/) (2023)
- [Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning](https://mlsanity.com/paper/2203.09137/) (2022)
- [Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?](https://mlsanity.com/paper/2203.14577/) (2022)
- [Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search](https://mlsanity.com/paper/2201.09785/) (2022)
- [Generalization Properties of NAS under Activation and Skip Connection Search](https://mlsanity.com/paper/2209.07238/) (2022)
- [EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search](https://mlsanity.com/paper/2210.06015/) (2022)
- [Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representations](https://mlsanity.com/paper/2410.13271/) (2024)
- [Robust NAS under adversarial training: benchmark, theory, and beyond](https://mlsanity.com/paper/2403.13134/) (2024)
- [ParZC: Parametric Zero-Cost Proxies for Efficient NAS](https://mlsanity.com/paper/2402.02105/) (2024)
- [CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework](https://mlsanity.com/paper/2406.01414/) (2024)
- [MONAS: Efficient Zero-Shot Neural Architecture Search for MCUs](https://mlsanity.com/paper/2408.15034/) (2024)
- [LayerNAS: Neural Architecture Search in Polynomial Complexity](https://mlsanity.com/paper/2304.11517/) (2023)
- [Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Models](https://mlsanity.com/paper/2405.07527/) (2024)
- [Analyzing Tree Architectures in Ensembles via Neural Tangent Kernel](https://mlsanity.com/paper/2205.12904/) (2022)
- [Differentiable Search of Accurate and Robust Architectures](https://mlsanity.com/paper/2212.14049/) (2022)
- [Carbon-Efficient Neural Architecture Search](https://mlsanity.com/paper/2307.04131/) (2023)
- [TG-NAS: Generalizable Zero-Cost Proxies with Operator Description Embedding and Graph Learning for Efficient Neural Architecture Search](https://mlsanity.com/paper/2404.00271/) (2024)
- [When Training-Free NAS Meets Vision Transformer: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2405.04536/) (2024)
- [Delving into Spectral Clustering with Vision-Language Representations](https://mlsanity.com/paper/2602.09586/) (2026)
- [Brain-inspired Evolutionary Architectures for Spiking Neural Networks](https://mlsanity.com/paper/2309.05263/) (2023)
- [One Search Fits All: Pareto-Optimal Eco-Friendly Model Selection](https://mlsanity.com/paper/2505.01468/) (2025)
- [Where and How to Enhance: Discovering Bit-Width Contribution for Mixed Precision Quantization](https://mlsanity.com/paper/2508.03002/) (2025)
- [Kernels, Data & Physics](https://mlsanity.com/paper/2307.02693/) (2023)
- [Efficient Transferability Assessment for Selection of Pre-trained Detectors](https://mlsanity.com/paper/2403.09432/) (2024)
- [Variation Matters: from Mitigating to Embracing Zero-Shot NAS Ranking Function Variation](https://mlsanity.com/paper/2502.19657/) (2025)

## Abstract (arXiv)

Many existing neural architecture search (NAS) solutions rely on downstream training for architecture evaluation, which takes enormous computations. Considering that these computations bring a large carbon footprint, this paper aims to explore a green (namely environmental-friendly) NAS solution that evaluates architectures without training. Intuitively, gradients, induced by the architecture itself, directly decide the convergence and generalization results. It motivates us to propose the gradient kernel hypothesis: Gradients can be used as a coarse-grained proxy of downstream training to evaluate random-initialized networks. To support the hypothesis, we conduct a theoretical analysis and find a practical gradient kernel that has good correlations with training loss and validation performance. According to this hypothesis, we propose a new kernel based architecture search approach KNAS. Experiments show that KNAS achieves competitive results with orders of magnitude faster than "train-then-test" paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. The searched network also outperforms strong baseline RoBERTA-large on two text classification tasks. Codes are available at urlhttps://github.com/Jingjing-NLP/KNAS .