# LLM-Guided Evolution: An Autonomous Model Optimization for Object Detection

arXiv: https://arxiv.org/abs/2504.02280 · HTML: https://mlsanity.com/paper/2504.02280/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 16 of 16)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [YOLOv12: Attention-Centric Real-Time Object Detectors](https://mlsanity.com/paper/2502.12524/) (2025)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap](https://mlsanity.com/paper/2401.10034/) (2024)
- [LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization](https://mlsanity.com/paper/2306.01102/) (2023)
- [DetNAS: Backbone Search for Object Detection](https://mlsanity.com/paper/1903.10979/) (2019)
- [DAMO-YOLO : A Report on Real-Time Object Detection Design](https://mlsanity.com/paper/2211.15444/) (2022)
- [SM-NAS: Structural-to-Modular Neural Architecture Search for Object Detection](https://mlsanity.com/paper/1911.09929/) (2019)
- [AlphaD3M: Machine Learning Pipeline Synthesis](https://mlsanity.com/paper/2111.02508/) (2021)
- [LLM Guided Evolution -- The Automation of Models Advancing Models](https://mlsanity.com/paper/2403.11446/) (2024)
- [YOLOv5, YOLOv8 and YOLOv10: The Go-To Detectors for Real-time Vision](https://mlsanity.com/paper/2407.02988/) (2024)
- [EAutoDet: Efficient Architecture Search for Object Detection](https://mlsanity.com/paper/2203.10747/) (2022)
- [A Novel lightweight Convolutional Neural Network, ExquisiteNetV2](https://mlsanity.com/paper/2105.09008/) (2021)

## Built on by (most cited first, 5 of 5)

- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [Cognitive-YOLO: LLM-Driven Architecture Synthesis from First Principles of Data for Object Detection](https://mlsanity.com/paper/2512.12281/) (2025)
- [GenAutoML: An Agentic Framework for Dynamic Architecture Generation and Optimization in Time-Series Analysis](https://mlsanity.com/paper/2606.05860/) (2026)
- [Vision Meets Language: A RAG-Augmented YOLOv8 Framework for Coffee Disease Diagnosis and Farmer Assistance](https://mlsanity.com/paper/2505.21544/) (2025)
- [A Hybrid Vision-Language Architecture for Automated Defect Reasoning and Report Generation in Industrial Inspection](https://mlsanity.com/paper/2605.26533/) (2026)

## Abstract (arXiv)

In machine learning, Neural Architecture Search (NAS) requires domain knowledge of model design and a large amount of trial-and-error to achieve promising performance. Meanwhile, evolutionary algorithms have traditionally relied on fixed rules and pre-defined building blocks. The Large Language Model (LLM)-Guided Evolution (GE) framework transformed this approach by incorporating LLMs to directly modify model source code for image classification algorithms on CIFAR data and intelligently guide mutations and crossovers. A key element of LLM-GE is the "Evolution of Thought" (EoT) technique, which establishes feedback loops, allowing LLMs to refine their decisions iteratively based on how previous operations performed. In this study, we perform NAS for object detection by improving LLM-GE to modify the architecture of You Only Look Once (YOLO) models to enhance performance on the KITTI dataset. Our approach intelligently adjusts the design and settings of YOLO to find the optimal algorithms against objective such as detection accuracy and speed. We show that LLM-GE produced variants with significant performance improvements, such as an increase in Mean Average Precision from 92.5% to 94.5%. This result highlights the flexibility and effectiveness of LLM-GE on real-world challenges, offering a novel paradigm for automated machine learning that combines LLM-driven reasoning with evolutionary strategies.