mlsanityData to 7 Oct 2026

KNAS: Green Neural Architecture Search

arXiv 2111.13293 ↗First posted 26 Nov 2021cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Many existing neural architecture search (NAS) solutions rely on downstream training for architecture evaluation, which takes enormous computations. Considering that these computations bring a large carbon footprint, this paper aims to explore a green (namely environmental-friendly) NAS solution that evaluates architectures without training. Intuitively, gradients, induced by the architecture itself, directly decide the convergence and generalization results. It motivates us to propose the gradient kernel hypothesis: Gradients can be used as a coarse-grained proxy of downstream training to evaluate random-initialized networks. To support the hypothesis, we conduct a theoretical analysis and find a practical gradient kernel that has good correlations with training loss and validation performance. According to this hypothesis, we propose a new kernel based architecture search approach KNAS. Experiments show that KNAS achieves competitive results with orders of magnitude faster than "train-then-test" paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. The searched network also outperforms strong baseline RoBERTA-large on two text classification tasks. Codes are available at urlhttps://github.com/Jingjing-NLP/KNAS .

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 53 of its 63 references that are in the map. First posted on arXiv 26 Nov 2021.

Builds on

53 of 63 references.

References · 53

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2016Layer Normalizationcited by 5,579
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  9. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  10. 2018Group Normalizationcited by 1,514
  11. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  12. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
38 more
  1. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  2. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  3. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  4. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  7. 2017Self-Normalizing Neural Networkscited by 591
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  10. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  11. 2017Progressive Neural Architecture Searchcited by 483
  12. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  13. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  14. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  15. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  16. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  17. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  18. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 277
  19. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  20. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  21. 2020Neural Architecture Search without Trainingcited by 241
  22. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  23. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  24. 2021Zero-Cost Proxies for Lightweight NAScited by 194
  25. 2019The Evolved Transformercited by 180
  26. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 176
  27. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 168
  28. 2013On the Convergence of Decentralized Gradient Descentcited by 158
  29. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  30. 2019Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Searchcited by 129
  31. 2015All you need is a good initcited by 124
  32. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  33. 2019DARTS+: Improved Differentiable Architecture Search with Early Stoppingcited by 105
  34. 2019SGAS: Sequential Greedy Architecture Searchcited by 69
  35. 2020GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernetcited by 69
  36. 2020EcoNAS: Finding Proxies for Economical Neural Architecture Searchcited by 61
  37. 2020MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulationcited by 56
  38. 2020Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Searchcited by 30

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 8; 2024: 10; 2025: 5; 2026: 2.

33 papers.

Sort:

By area:

  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  2. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  3. 2022Neural Architecture Search for Spiking Neural Networkscited by 52
  4. 2024Quantum Architecture Search: A Surveycited by 31
  5. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  6. 2023LEMON: Lossless model expansioncited by 23
  7. 2022Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learningcited by 22
  8. 2022Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?cited by 18
  9. 2022Unifying and Boosting Gradient-Based Training-Free Neural Architecture Searchcited by 17
  10. 2022Generalization Properties of NAS under Activation and Skip Connection Searchcited by 14
  11. 2022EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Searchcited by 11
  12. 2024Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representationscited by 8
21 more
  1. 2024Robust NAS under adversarial training: benchmark, theory, and beyondcited by 6
  2. 2024ParZC: Parametric Zero-Cost Proxies for Efficient NAScited by 5
  3. 2024CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Frameworkcited by 5
  4. 2024MONAS: Efficient Zero-Shot Neural Architecture Search for MCUscited by 4
  5. 2023LayerNAS: Neural Architecture Search in Polynomial Complexitycited by 3
  6. 2024Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Modelscited by 3
  7. 2022Analyzing Tree Architectures in Ensembles via Neural Tangent Kernelcited by 2
  8. 2022Differentiable Search of Accurate and Robust Architecturescited by 2
  9. 2023Carbon-Efficient Neural Architecture Searchcited by 2
  10. 2024TG-NAS: Generalizable Zero-Cost Proxies with Operator Description Embedding and Graph Learning for Efficient Neural Architecture Searchcited by 2
  11. 2024When Training-Free NAS Meets Vision Transformer: A Neural Tangent Kernel Perspectivecited by 2
  12. 2026Delving into Spectral Clustering with Vision-Language Representationscited by 2
  13. 2023Brain-inspired Evolutionary Architectures for Spiking Neural Networkscited by 1
  14. 2025One Search Fits All: Pareto-Optimal Eco-Friendly Model Selectioncited by 1
  15. 2025Where and How to Enhance: Discovering Bit-Width Contribution for Mixed Precision Quantizationcited by 1
  16. 2023Kernels, Data & Physicscited by 0
  17. 2024Efficient Transferability Assessment for Selection of Pre-trained Detectorscited by 0
  18. 2025Variation Matters: from Mitigating to Embracing Zero-Shot NAS Ranking Function Variationcited by 0
  19. 2025Is Supervised Learning Really That Different from Unsupervised?cited by 0
  20. 2025Half Search Space is All You Needcited by 0
  21. 2026Designing Sustainable Federated Learning as a Service using Neural Architecture Searchcited by 0