mlsanityData to 7 Oct 2026

When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges

arXiv 2401.10510 ↗First posted 19 Jan 2024cs.NE cs.AI cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Pre-trained large language models (LLMs) exhibit powerful capabilities for generating natural text. Evolutionary algorithms (EAs) can discover diverse solutions to complex real-world problems. Motivated by the common collective and directionality of text generation and evolution, this paper first illustrates the conceptual parallels between LLMs and EAs at a micro level, which includes multiple one-to-one key characteristics: token representation and individual representation, position encoding and fitness shaping, position embedding and selection, Transformers block and reproduction, and model training and parameter adaptation. These parallels highlight potential opportunities for technical advancements in both LLMs and EAs. Subsequently, we analyze existing interdisciplinary research from a macro perspective to uncover critical challenges, with a particular focus on evolutionary fine-tuning and LLM-enhanced EAs. These analyses not only provide insights into the evolutionary mechanisms behind LLMs but also offer potential directions for enhancing the capabilities of artificial agents.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 64 of its 112 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

64 of 112 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2023Large Language Models as Optimizerscited by 830
  10. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  11. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  12. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
38 more
  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  2. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  3. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  4. 2023Role-Play with Large Language Modelscited by 290
  5. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  6. 2019Pareto Multi-Task Learningcited by 224
  7. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  8. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  9. 2022Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editingcited by 165
  10. 2019Adaptively Sparse Transformerscited by 154
  11. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  12. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  13. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  14. 2023Large Language Models as Evolutionary Optimizerscited by 122
  15. 2023Language Model Crossover: Variation through Few-Shot Promptingcited by 119
  16. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  17. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 88
  18. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  19. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  20. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 85
  21. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 80
  22. 2023Can GPT-4 Perform Neural Architecture Search?cited by 74
  23. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  24. 2019TENER: Adapting Transformer Encoder for Named Entity Recognitioncited by 69
  25. 2023MarioGPT: Open-Ended Text2Level Generation through Large Language Modelscited by 63
  26. 2022BBTv2: Towards a Gradient-Free Future with Large Language Modelscited by 53
  27. 2019Greed is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisationcited by 47
  28. 2022Discovering Evolution Strategies via Meta-Black-Box Optimizationcited by 44
  29. 2023Quality-Diversity through AI Feedbackcited by 40
  30. 2023Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimizationcited by 33
  31. 2023Large Language Model for Multi-objective Evolutionary Optimizationcited by 29
  32. 2023Gradient-Free Textual Inversioncited by 28
  33. 2023FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Modelscited by 25
  34. 2023Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Modelscited by 21
  35. 2021Analogous to Evolutionary Algorithm: Designing a Unified Sequence Modelcited by 20
  36. 2023ChatGPT and Other Large Language Models as Evolutionary Engines for Online Interactive Collaborative Game Designcited by 20
  37. 2023Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruningcited by 17
  38. 2023B2Opt: Learning to Optimize Black-box Optimization with Little Budgetcited by 15

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 8; 2026: 6.

20 papers.

Sort:

By area:

  1. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  2. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  3. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 28
  4. 2024Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithmscited by 12
  5. 2024HM3: Hierarchical Multi-Objective Model Merging for Pretrained Modelscited by 8
  6. 2025SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithmscited by 7
  7. 2025Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systemscited by 6
  8. 2024Identify Critical Nodes in Complex Network with Large Language Modelscited by 4
  9. 2025Large Language Models in Operations Research: Methods, Applications, and Challengescited by 4
  10. 2025Can Large Language Models Be Trusted as Evolutionary Optimizers for Network-Structured Combinatorial Problems?cited by 3
  11. 2026Task-free Adaptive Meta Black-box Optimizationcited by 2
  12. 2025Learning Evolution via Optimization Knowledge Adaptationcited by 1
8 more
  1. 2025EvoSpeak: Large Language Models for Interpretable Genetic Programming-Evolved Heuristicscited by 1
  2. 2026Evolutionary Intelligence for Scientific Discovery: From Evolutionary Computation to Cumulative Discovery Systemscited by 1
  3. 2024An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problemscited by 0
  4. 2025An In-depth Study of LLM Contributions to the Bin Packing Problemcited by 0
  5. 2026From Prompt to Protocol: Fast Charging Batteries with Large Language Modelscited by 0
  6. 2026EvoGens: A Population-Based Heuristic Search Framework for Scientific Idea Generationcited by 0
  7. 2026Model Merging to Evolution: Parameter Space Exploration for Expert Modelscited by 0
  8. 2026Hyper Algorithm Design Agent: Evolving Learnable Optimizer from Zerocited by 0