# Can GPT-4 Perform Neural Architecture Search?

arXiv: https://arxiv.org/abs/2304.10970 · HTML: https://mlsanity.com/paper/2304.10970/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 82 references that are in the map.
- First posted on arXiv 21 Apr 2023.

## Builds on (most cited first, 30 of 66)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)

## Built on by (most cited first, 30 of 74)

- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers](https://mlsanity.com/paper/2309.08532/) (2023)
- [LLM-SR: Scientific Equation Discovery via Programming with Large Language Models](https://mlsanity.com/paper/2404.18400/) (2024)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities](https://mlsanity.com/paper/2307.16376/) (2023)
- [Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap](https://mlsanity.com/paper/2401.10034/) (2024)
- [AutoML-GPT: Automatic Machine Learning with GPT](https://mlsanity.com/paper/2305.02499/) (2023)
- [Using Large Language Models for Hyperparameter Optimization](https://mlsanity.com/paper/2312.04528/) (2023)
- [GPT4GEO: How a Language Model Sees the World's Geography](https://mlsanity.com/paper/2306.00020/) (2023)
- [MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks](https://mlsanity.com/paper/2304.14979/) (2023)
- [Algorithm Evolution Using Large Language Model](https://mlsanity.com/paper/2311.15249/) (2023)
- [LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizers](https://mlsanity.com/paper/2503.14434/) (2025)
- [Language Models as Black-Box Optimizers for Vision-Language Models](https://mlsanity.com/paper/2309.05950/) (2023)
- [AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks](https://mlsanity.com/paper/2306.08107/) (2023)
- [Large Language Model for Multi-objective Evolutionary Optimization](https://mlsanity.com/paper/2310.12541/) (2023)
- [LLM Performance Predictors are good initializers for Architecture Search](https://mlsanity.com/paper/2310.16712/) (2023)
- [Large Language Model-Based Evolutionary Optimizer: Reasoning with elitism](https://mlsanity.com/paper/2403.02054/) (2024)
- [FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch](https://mlsanity.com/paper/2406.04824/) (2024)
- [When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges](https://mlsanity.com/paper/2401.10510/) (2024)
- [Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey](https://mlsanity.com/paper/2411.10478/) (2024)
- [AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks](https://mlsanity.com/paper/2508.00890/) (2025)
- [AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks](https://mlsanity.com/paper/2402.15351/) (2024)
- [Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection](https://mlsanity.com/paper/2307.13529/) (2023)
- [When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs](https://mlsanity.com/paper/2309.14488/) (2023)
- [LLEMA: Evolutionary Search with LLMs for Multi-Objective Materials Discovery](https://mlsanity.com/paper/2510.22503/) (2025)
- [On the Viability of using LLMs for SW/HW Co-Design: An Example in Designing CiM DNN Accelerators](https://mlsanity.com/paper/2306.06923/) (2023)
- [LeMo-NADe: Multi-Parameter Neural Architecture Discovery with LLMs](https://mlsanity.com/paper/2402.18443/) (2024)
- [FL-NAS: Towards Fairness of NAS for Resource Constrained Devices via Large Language Models](https://mlsanity.com/paper/2402.06696/) (2024)
- [Large Language Models Synergize with Automated Machine Learning](https://mlsanity.com/paper/2405.03727/) (2024)
- [NADER: Neural Architecture Design via Multi-Agent Collaboration](https://mlsanity.com/paper/2412.19206/) (2024)

## Abstract (arXiv)

We investigate the potential of GPT-4~citegpt4 to perform Neural Architecture Search (NAS) -- the task of designing effective neural architectures. Our proposed approach, GPT-4 Enhanced Neural archItectUre Search (GENIUS), leverages the generative capabilities of GPT-4 as a black-box optimiser to quickly navigate the architecture search space, pinpoint promising candidates, and iteratively refine these candidates to improve performance. We assess GENIUS across several benchmarks, comparing it with existing state-of-the-art NAS techniques to illustrate its effectiveness. Rather than targeting state-of-the-art performance, our objective is to highlight GPT-4's potential to assist research on a challenging technical problem through a simple prompting scheme that requires relatively limited domain expertisefootnoteCode available at hrefhttps://github.com/mingkai-zheng/GENIUShttps://github.com/mingkai-zheng/GENIUS.. More broadly, we believe our preliminary results point to future research that harnesses general purpose language models for diverse optimisation tasks. We also highlight important limitations to our study, and note implications for AI safety.