mlsanityData to 7 Oct 2026

Can GPT-4 Perform Neural Architecture Search?

arXiv 2304.10970 ↗First posted 21 Apr 2023cs.LGcited by 74

Not on the map. Why?

Abstract (arXiv)

We investigate the potential of GPT-4~citegpt4 to perform Neural Architecture Search (NAS) -- the task of designing effective neural architectures. Our proposed approach, GPT-4 Enhanced Neural archItectUre Search (GENIUS), leverages the generative capabilities of GPT-4 as a black-box optimiser to quickly navigate the architecture search space, pinpoint promising candidates, and iteratively refine these candidates to improve performance. We assess GENIUS across several benchmarks, comparing it with existing state-of-the-art NAS techniques to illustrate its effectiveness. Rather than targeting state-of-the-art performance, our objective is to highlight GPT-4's potential to assist research on a challenging technical problem through a simple prompting scheme that requires relatively limited domain expertisefootnoteCode available at hrefhttps://github.com/mingkai-zheng/GENIUShttps://github.com/mingkai-zheng/GENIUS.. More broadly, we believe our preliminary results point to future research that harnesses general purpose language models for diverse optimisation tasks. We also highlight important limitations to our study, and note implications for AI safety.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 66 of its 82 references that are in the map. First posted on arXiv 21 Apr 2023.

Builds on

66 of 82 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2016Densely Connected Convolutional Networkscited by 6,691
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  12. 2017Squeeze-and-Excitation Networkscited by 4,763
38 more
  1. 2014Going Deeper with Convolutionscited by 4,231
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2019Searching for MobileNetV3cited by 1,984
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  9. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  10. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  11. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  12. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  13. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  14. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  15. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  16. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  17. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  18. 2022Large Language Models Can Self-Improvecited by 656
  19. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  20. 2020ResNeSt: Split-Attention Networkscited by 445
  21. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  22. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  23. 2017Large-Scale Evolution of Image Classifierscited by 348
  24. 2018SNAS: Stochastic Neural Architecture Searchcited by 288
  25. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  26. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  27. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
  28. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  29. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  30. 2019FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Searchcited by 129
  31. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  32. 2023Emergent autonomous scientific research capabilities of large language modelscited by 127
  33. 2019MixConv: Mixed Depthwise Convolutional Kernelscited by 108
  34. 2019Blockwisely Supervised Neural Architecture Search with Knowledge Distillationcited by 81
  35. 2020DSNAS: Direct Neural Architecture Search without Parameter Retrainingcited by 69
  36. 2020GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernetcited by 69
  37. 2021BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Searchcited by 63
  38. 2022β-DARTS: Beta-Decay Regularization for Differentiable Architecture Searchcited by 63

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 16; 2025: 21; 2026: 16.

74 papers.

Sort:

By area:

  1. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  2. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  3. 2024LLM-SR: Scientific Equation Discovery via Programming with Large Language Modelscited by 117
  4. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  5. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  6. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  7. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 80
  8. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  9. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  10. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  11. 2023Algorithm Evolution Using Large Language Modelcited by 52
  12. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
38 more
  1. 2025LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizerscited by 35
  2. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  3. 2023Large Language Model for Multi-objective Evolutionary Optimizationcited by 29
  4. 2023LLM Performance Predictors are good initializers for Architecture Searchcited by 29
  5. 2024Large Language Model-Based Evolutionary Optimizer: Reasoning with elitismcited by 27
  6. 2024FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearchcited by 22
  7. 2024When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challengescited by 20
  8. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  9. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Taskscited by 16
  10. 2024AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Taskscited by 14
  11. 2023Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detectioncited by 13
  12. 2023When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTscited by 13
  13. 2023On the Viability of using LLMs for SW/HW Co-Design: An Example in Designing CiM DNN Acceleratorscited by 11
  14. 2024LeMo-NADe: Multi-Parameter Neural Architecture Discovery with LLMscited by 11
  15. 2025LLEMA: Evolutionary Search with LLMs for Multi-Objective Materials Discoverycited by 11
  16. 2024FL-NAS: Towards Fairness of NAS for Resource Constrained Devices via Large Language Modelscited by 9
  17. 2024Large Language Models Synergize with Automated Machine Learningcited by 8
  18. 2024NADER: Neural Architecture Design via Multi-Agent Collaborationcited by 8
  19. 2025Mimicking the Physicist's Eye:A VLM-centric Approach for Physics Formula Discoverycited by 6
  20. 2025CoLLM-NAS: Collaborative Large Language Models for Efficient Knowledge-Guided Neural Architecture Searchcited by 6
  21. 2026Closed-Loop LLM Discovery of Non-Standard Channel Priors in Vision Modelscited by 6
  22. 2023Exploring the Limits of ChatGPT in Software Security Applicationscited by 5
  23. 2025Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?cited by 5
  24. 2025LLM-Guided Search for Deletion-Correcting Codescited by 4
  25. 2025Generalizing Test-time Compute-optimal Scaling as an Optimizable Graphcited by 4
  26. 2025LLM4GNAS: A Large Language Model Based Toolkit for Graph Neural Architecture Searchcited by 3
  27. 2025LLMs as In-Context Meta-Learners for Model and Hyperparameter Selectioncited by 3
  28. 2026Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffscited by 3
  29. 2024Large Language Models for Tuning Evolution Strategiescited by 2
  30. 2024Adaptive Augmentation Policy Optimization with LLM Feedbackcited by 2
  31. 2025MoColl: Agent-Based Specific and General Model Collaboration for Image Captioningcited by 2
  32. 2025PhaseNAS: Language-Model Driven Architecture Search with Dynamic Phase Adaptationcited by 2
  33. 2025MetaLLMix : An XAI Aided LLM-Meta-learning Based Approach for Hyper-parameters Optimizationcited by 2
  34. 2025FELA: A Multi-Agent Evolutionary System for Feature Engineering of Industrial Event Log Datacited by 2
  35. 2026Structured Progressive Knowledge Activation for LLM-Driven Neural Architecture Searchcited by 2
  36. 2026An Agentic AI Scientific Community for Automated Neural Operator Discoverycited by 2
  37. 2023arXiVeri: Automatic table verification with GPTcited by 1
  38. 2023Transforming the Output of Generative Pre-trained Transformer: The Influence of the PGI Framework on Attention Dynamicscited by 1

50 most cited of 74 All 74 (JSON)