# Algorithm Discovery With LLMs: Evolutionary Search Meets Reinforcement Learning

arXiv: https://arxiv.org/abs/2504.05108 · HTML: https://mlsanity.com/paper/2504.05108/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 49 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 11 of 11)

- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2406.05370/) (2024)
- [PromptTTS: Controllable Text-to-Speech with Text Descriptions](https://mlsanity.com/paper/2211.12171/) (2022)
- [Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models](https://mlsanity.com/paper/2304.09337/) (2023)
- [PromptTTS 2: Describing and Generating Voices with Text Prompt](https://mlsanity.com/paper/2309.02285/) (2023)
- [Believing Anthropomorphism: Examining the Role of Anthropomorphic Cues on Trust in Large Language Models](https://mlsanity.com/paper/2405.06079/) (2024)
- [PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions](https://mlsanity.com/paper/2305.19522/) (2023)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [VoiceCoach: Interactive Evidence-based Training for Voice Modulation Skills in Public Speaking](https://mlsanity.com/paper/2001.07876/) (2020)

## Built on by (most cited first, 30 of 46)

- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [KL-Regularized Reinforcement Learning is Designed to Mode Collapse](https://mlsanity.com/paper/2510.20817/) (2025)
- [MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI](https://mlsanity.com/paper/2605.08678/) (2026)
- [HiFo-Prompt: Prompting with Hindsight and Foresight for LLM-based Automatic Heuristic Design](https://mlsanity.com/paper/2508.13333/) (2025)
- [Training AI Scientists to Replicate Research](https://mlsanity.com/paper/2608.13331/) (2026)
- [MiGrATe: Mixed-Policy GRPO for Adaptation at Test-Time](https://mlsanity.com/paper/2508.08641/) (2025)
- [VRPAgent: LLM-Driven Discovery of Heuristic Operators for Vehicle Routing Problems](https://mlsanity.com/paper/2510.07073/) (2025)
- [LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression](https://mlsanity.com/paper/2505.18602/) (2025)
- [Fine-tuning Large Language Model for Automated Algorithm Design](https://mlsanity.com/paper/2507.10614/) (2025)
- [Large Language Models in Operations Research: Methods, Applications, and Challenges](https://mlsanity.com/paper/2509.18180/) (2025)
- [Evolutionary System Prompt Learning for Reinforcement Learning in LLMs](https://mlsanity.com/paper/2602.14697/) (2026)
- [What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search](https://mlsanity.com/paper/2604.19440/) (2026)
- [AutoEP: LLMs-Driven Automation of Hyperparameter Evolution for Metaheuristic Algorithms](https://mlsanity.com/paper/2509.23189/) (2025)
- [GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learning](https://mlsanity.com/paper/2604.02721/) (2026)
- [TIDE: Tuning-Integrated Dynamic Evolution for LLM-Based Automated Heuristic Design](https://mlsanity.com/paper/2601.21239/) (2026)
- [AHD Agent: Agentic Reinforcement Learning for Automatic Heuristic Design](https://mlsanity.com/paper/2605.08756/) (2026)
- [Game-Theoretic Co-Evolution for LLM-Based Heuristic Discovery](https://mlsanity.com/paper/2601.22896/) (2026)
- [Heuresis: Search Strategies for Autonomous AI Research Agents Across Quality, Diversity and Novelty](https://mlsanity.com/paper/2606.25198/) (2026)
- [GeoEvolve: Automating Geospatial Model Discovery via Multi-Agent Large Language Models](https://mlsanity.com/paper/2509.21593/) (2025)
- [MCCE: A Framework for Multi-LLM Collaborative Search in Discrete Spaces with Similarity-Filtered Preference Learning](https://mlsanity.com/paper/2510.06270/) (2025)
- [Automated near-term quantum algorithm discovery for molecular ground states](https://mlsanity.com/paper/2603.26359/) (2026)
- [PyVRP^+: LLM-Driven Metacognitive Heuristic Evolution for Hybrid Genetic Search in Vehicle Routing Problems](https://mlsanity.com/paper/2604.07872/) (2026)
- [BEAM: Bi-level Memory-adaptive Algorithmic Evolution for LLM-Powered Heuristic Design](https://mlsanity.com/paper/2604.12898/) (2026)
- [Back to the Beginning of Heuristic Design: Bridging Code and Knowledge with LLMs](https://mlsanity.com/paper/2605.06123/) (2026)
- [Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization](https://mlsanity.com/paper/2605.19782/) (2026)
- [COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learning](https://mlsanity.com/paper/2608.30109/) (2026)
- [Data-Driven Discovery of Interpretable Kalman Filter Variants through Large Language Models and Genetic Programming](https://mlsanity.com/paper/2508.11703/) (2025)
- [Lark: Biologically Inspired Neuroevolution for Multi-Stakeholder LLM Agents](https://mlsanity.com/paper/2510.16978/) (2025)
- [ATLAS: A Multi-LLM Training Framework for EvoDPO with Adaptive Reference Evolution](https://mlsanity.com/paper/2602.02709/) (2026)
- [ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution](https://mlsanity.com/paper/2603.02510/) (2026)

## Abstract (arXiv)

Discovering efficient algorithms for solving complex problems has been an outstanding challenge in mathematics and computer science, requiring substantial human expertise over the years. Recent advancements in evolutionary search with large language models (LLMs) have shown promise in accelerating the discovery of algorithms across various domains, particularly in mathematics and optimization. However, existing approaches treat the LLM as a static generator, missing the opportunity to update the model with the signal obtained from evolutionary exploration. In this work, we propose to augment LLM-based evolutionary search by continuously refining the search operator - the LLM - through reinforcement learning (RL) fine-tuning. Our method leverages evolutionary search as an exploration strategy to discover improved algorithms, while RL optimizes the LLM policy based on these discoveries. Our experiments on combinatorial optimization tasks demonstrate that integrating RL with evolutionary search accelerates the discovery of superior algorithms, showcasing the potential of RL-enhanced evolutionary strategies for algorithm design.