# When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges

arXiv: https://arxiv.org/abs/2401.10510 · HTML: https://mlsanity.com/paper/2401.10510/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 112 references that are in the map.
- First posted on arXiv 19 Jan 2024.

## Builds on (most cited first, 30 of 64)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023)
- [Black-Box Tuning for Language-Model-as-a-Service](https://mlsanity.com/paper/2201.03514/) (2022)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing](https://mlsanity.com/paper/2212.10789/) (2022)
- [Adaptively Sparse Transformers](https://mlsanity.com/paper/1909.00015/) (2019)
- [Open Sesame! Universal Black Box Jailbreaking of Large Language Models](https://mlsanity.com/paper/2309.01446/) (2023)
- [EvoPrompting: Language Models for Code-Level Neural Architecture Search](https://mlsanity.com/paper/2302.14838/) (2023)
- [GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://mlsanity.com/paper/2203.07281/) (2022)
- [Large Language Models as Evolutionary Optimizers](https://mlsanity.com/paper/2310.19046/) (2023)
- [Language Model Crossover: Variation through Few-Shot Prompting](https://mlsanity.com/paper/2302.12170/) (2023)
- [Towards artificial general intelligence via a multimodal foundation model](https://mlsanity.com/paper/2110.14378/) (2021)
- [Low-Rank Bottleneck in Multi-head Attention Models](https://mlsanity.com/paper/2002.07028/) (2020)
- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)

## Built on by (most cited first, 20 of 20)

- [LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation](https://mlsanity.com/paper/2403.01131/) (2024)
- [Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models](https://mlsanity.com/paper/2412.02980/) (2024)
- [Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning](https://mlsanity.com/paper/2509.24372/) (2025)
- [Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithms](https://mlsanity.com/paper/2410.20848/) (2024)
- [HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models](https://mlsanity.com/paper/2409.18893/) (2024)
- [SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms](https://mlsanity.com/paper/2506.06499/) (2025)
- [Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systems](https://mlsanity.com/paper/2505.15741/) (2025)
- [Large Language Models in Operations Research: Methods, Applications, and Challenges](https://mlsanity.com/paper/2509.18180/) (2025)
- [Identify Critical Nodes in Complex Network with Large Language Models](https://mlsanity.com/paper/2403.03962/) (2024)
- [Can Large Language Models Be Trusted as Evolutionary Optimizers for Network-Structured Combinatorial Problems?](https://mlsanity.com/paper/2501.15081/) (2025)
- [Task-free Adaptive Meta Black-box Optimization](https://mlsanity.com/paper/2601.21475/) (2026)
- [Learning Evolution via Optimization Knowledge Adaptation](https://mlsanity.com/paper/2501.02200/) (2025)
- [EvoSpeak: Large Language Models for Interpretable Genetic Programming-Evolved Heuristics](https://mlsanity.com/paper/2510.02686/) (2025)
- [EvoGens: A Population-Based Heuristic Search Framework for Scientific Idea Generation](https://mlsanity.com/paper/2605.30961/) (2026)
- [Evolutionary Intelligence for Scientific Discovery: From Evolutionary Computation to Cumulative Discovery Systems](https://mlsanity.com/paper/2607.09025/) (2026)
- [An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems](https://mlsanity.com/paper/2501.00562/) (2024)
- [An In-depth Study of LLM Contributions to the Bin Packing Problem](https://mlsanity.com/paper/2510.27353/) (2025)
- [From Prompt to Protocol: Fast Charging Batteries with Large Language Models](https://mlsanity.com/paper/2601.09626/) (2026)
- [Model Merging to Evolution: Parameter Space Exploration for Expert Models](https://mlsanity.com/paper/2606.28373/) (2026)
- [Hyper Algorithm Design Agent: Evolving Learnable Optimizer from Zero](https://mlsanity.com/paper/2609.35328/) (2026)

## Abstract (arXiv)

Pre-trained large language models (LLMs) exhibit powerful capabilities for generating natural text. Evolutionary algorithms (EAs) can discover diverse solutions to complex real-world problems. Motivated by the common collective and directionality of text generation and evolution, this paper first illustrates the conceptual parallels between LLMs and EAs at a micro level, which includes multiple one-to-one key characteristics: token representation and individual representation, position encoding and fitness shaping, position embedding and selection, Transformers block and reproduction, and model training and parameter adaptation. These parallels highlight potential opportunities for technical advancements in both LLMs and EAs. Subsequently, we analyze existing interdisciplinary research from a macro perspective to uncover critical challenges, with a particular focus on evolutionary fine-tuning and LLM-enhanced EAs. These analyses not only provide insights into the evolutionary mechanisms behind LLMs but also offer potential directions for enhancing the capabilities of artificial agents.