mlsanityData to 7 Oct 2026

Neural Architecture Search: Insights from 1000 Papers

arXiv 2301.08727 ↗First posted 20 Jan 2023cs.LG cs.AI stat.MLcited by 136

Not on the map. Why?

Abstract (arXiv)

In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, and reinforcement learning. Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas. Neural architecture search (NAS), the process of automating the design of neural architectures for a given task, is an inevitable next step in automating machine learning and has already outpaced the best human-designed architectures on many tasks. In the past few years, research in NAS has been progressing rapidly, with over 1000 papers released since 2020 (Deng and Lindauer, 2021). In this survey, we provide an organized and comprehensive guide to neural architecture search. We give a taxonomy of search spaces, algorithms, and speedup techniques, and we discuss resources such as benchmarks, best practices, other surveys, and open-source libraries.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Builds on 243 of its 320 references that are in the map. First posted on arXiv 20 Jan 2023.

Builds on

243 of 320 references.

References · 243

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2017Proximal Policy Optimization Algorithmscited by 14,441
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2016Densely Connected Convolutional Networkscited by 6,691
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
38 more
  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2016Wide Residual Networkscited by 3,057
  9. 2016Improved Techniques for Training GANscited by 2,938
  10. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  11. 2017Improved Training of Wasserstein GANscited by 2,223
  12. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  13. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  14. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  15. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  16. 2017Deep Setscited by 1,394
  17. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  18. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  19. 2020Designing Network Design Spacescited by 954
  20. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  21. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  22. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  23. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  24. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  25. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  26. 2014Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognitioncited by 719
  27. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  28. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  29. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  30. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  31. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  32. 2015Listen, Attend and Spellcited by 542
  33. 2017Progressive Neural Architecture Searchcited by 483
  34. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  35. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  36. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  37. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  38. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412

50 most cited of 243 All 243 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 45; 2025: 43; 2026: 28.

136 papers.

Sort:

By area:

  1. 2025AIDE: AI-Driven Exploration in the Space of Codecited by 206
  2. 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
  3. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 106
  4. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  5. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 57
  6. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 49
  7. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  8. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  9. 2024Quantum Architecture Search: A Surveycited by 31
  10. 2023Artificial intelligence to advance Earth observation: : A review of models, recent trends, and pathways forwardcited by 28
  11. 2024Universal Time-Series Representation Learning: A Surveycited by 28
  12. 2024Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performancecited by 22
38 more
  1. 2023Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoMLcited by 16
  2. 2024A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trendscited by 16
  3. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  4. 2024PaSTe: Improving the Efficiency of Visual Anomaly Detection at the Edgecited by 16
  5. 2023DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Modelscited by 15
  6. 2024Position: A Call to Action for a Human-Centered AutoML Paradigmcited by 14
  7. 2024Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Modelscited by 13
  8. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  9. 2024SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAScited by 11
  10. 2024Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selectioncited by 11
  11. 2025Astraea: A Token-wise Acceleration Framework for Video Diffusion Transformerscited by 11
  12. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  13. 2024Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapescited by 10
  14. 2025Regression Language Models for Codecited by 10
  15. 2025Enhancing LLM-Based Neural Network Generation: Few-Shot Prompting and Efficient Validation for Automated Architecture Designcited by 10
  16. 2026From Memorization to Creativity: LLM as a Designer of Novel Neural Architecturescited by 10
  17. 2023PLiNIO: A User-Friendly Library of Gradient-based Methods for Complexity-aware DNN Optimizationcited by 9
  18. 2023Bridging the Gap Between Foundation Models and Heterogeneous Federated Learningcited by 9
  19. 2024Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignmentcited by 9
  20. 2023Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devicescited by 7
  21. 2024STAR: Synthesis of Tailored Architecturescited by 7
  22. 2025Composer: A Search Framework for Hybrid Neural Architecture Designcited by 7
  23. 2026Resource-Efficient Iterative LLM-Based NAS with Feedback Memorycited by 7
  24. 2023An algorithmic framework for the optimization of deep neural networks architectures and hyperparameterscited by 6
  25. 2023Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyMLcited by 6
  26. 2024Robust NAS under adversarial training: benchmark, theory, and beyondcited by 6
  27. 2024Joint Pruning and Channel-wise Mixed-Precision Quantization for Efficient Deep Neural Networkscited by 6
  28. 2025Improving LLM-based Global Optimization with Search Space Partitioningcited by 6
  29. 2024Future Directions in the Theory of Graph Machine Learningcited by 5
  30. 2024Insights from the Use of Previously Unseen Neural Architecture Search Datasetscited by 5
  31. 2024einspace: Searching for Neural Architectures from Fundamental Operationscited by 5
  32. 2024Scalable Reinforcement Learning-based Neural Architecture Searchcited by 5
  33. 2024MATCH: Model-Aware TVM-based Compilation for Heterogeneous Edge Devicescited by 5
  34. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  35. 2025Transferrable Surrogates in Expressive Neural Architecture Search Spacescited by 5
  36. 2023HyperPPO: A scalable method for finding small policies for robotic controlcited by 4
  37. 2023IncepSE: Leveraging InceptionTime's performance with Squeeze and Excitation mechanism in ECG analysiscited by 4
  38. 2023Weight-Entanglement Meets Gradient-Based Neural Architecture Searchcited by 4

50 most cited of 136 All 136 (JSON)