mlsanityData to 7 Oct 2026

Large Language Model-Based Evolutionary Optimizer: Reasoning with elitism

arXiv 2403.02054 ↗First posted 4 Mar 2024cs.AIcited by 27

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable reasoning abilities, prompting interest in their application as black-box optimizers. This paper asserts that LLMs possess the capability for zero-shot optimization across diverse scenarios, including multi-objective and high-dimensional problems. We introduce a novel population-based method for numerical optimization using LLMs called Language-Model-Based Evolutionary Optimizer (LEO). Our hypothesis is supported through numerical examples, spanning benchmark and industrial engineering problems such as supersonic nozzle shape optimization, heat transfer, and windfarm layout optimization. We compare our method to several gradient-based and gradient-free optimization approaches. While LLMs yield comparable results to state-of-the-art methods, their imaginative nature and propensity to hallucinate demand careful handling. We provide practical guidelines for obtaining reliable answers from LLMs and discuss method limitations and potential research directions.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 39 of its 64 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

39 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
26 more
  1. 2023Large Language Models as Optimizerscited by 830
  2. 2022A Generalist Agentcited by 802
  3. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  4. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  5. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  6. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  7. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  8. 2016Learning to reinforcement learncited by 369
  9. 2019Deep Learning for Symbolic Mathematicscited by 266
  10. 2020pymoo: Multi-objective Optimization in Pythoncited by 196
  11. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  12. 2023Large Language Models as Evolutionary Optimizerscited by 122
  13. 2021SymbolicGPT: A Generative Transformer Model for Symbolic Regressioncited by 101
  14. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  15. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 80
  16. 2023Tackling the Curse of Dimensionality with Physics-Informed Neural Networkscited by 78
  17. 2023Can GPT-4 Perform Neural Architecture Search?cited by 74
  18. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  19. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  20. 2021Linear algebra with transformerscited by 62
  21. 2023Algorithm Evolution Using Large Language Modelcited by 52
  22. 2023OptiMUS: Optimization Modeling Using MIP Solvers and large language modelscited by 39
  23. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  24. 2023Towards Optimizing with Large Language Modelscited by 34
  25. 2023Large Language Model for Multi-objective Evolutionary Optimizationcited by 29
  26. 2021Analyzing the Nuances of Transformers' Polynomial Simplification Abilitiescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 13; 2026: 6.

27 papers.

Sort:

By area:

  1. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 65
  2. 2025Evolving Deeper LLM Thinkingcited by 58
  3. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  4. 2024Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimizationcited by 23
  5. 2024Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithmscited by 12
  6. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  7. 2025A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solvingcited by 9
  8. 2024Using Large Language Models for Parametric Shape Optimizationcited by 7
  9. 2025ExLLM: Experience-Enhanced LLM Optimization for Molecular Design and Beyondcited by 6
  10. 2024QUBE: Enhancing Automatic Heuristic Design via Quality-Uncertainty Balanced Evolutioncited by 5
  11. 2025Vulnerability of Text-to-Image Models to Prompt Template Stealing: A Differential Evolution Approachcited by 5
  12. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
15 more
  1. 2025LLM-Guided Search for Deletion-Correcting Codescited by 4
  2. 2026What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Searchcited by 4
  3. 2025Can Large Language Models Be Trusted as Evolutionary Optimizers for Network-Structured Combinatorial Problems?cited by 3
  4. 2024Large Language Models for Tuning Evolution Strategiescited by 2
  5. 2025From Understanding to Excelling: Template-Free Algorithm Design through Structural-Functional Co-Evolutioncited by 2
  6. 2026Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombinationcited by 2
  7. 2025REMoH: A Reflective Evolution of Multi-objective Heuristics approach via Large Language Modelscited by 1
  8. 2025SOLID: a Framework of Synergizing Optimization and LLMs for Intelligent Decision-Makingcited by 1
  9. 2026DrugR: Optimizing Molecular Drugs through LLM-based Explicit Reasoningcited by 1
  10. 2024Large Language Models for Knowledge-Free Network Management: Feasibility Study and Opportunitiescited by 0
  11. 2024An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problemscited by 0
  12. 2025RoCo: Role-Based LLMs Collaboration for Automatic Heuristic Designcited by 0
  13. 2026MAFIG: Multi-agent Driven Formal Instruction Generation Frameworkcited by 0
  14. 2026ADKO: Agentic Decentralized Knowledge Optimizationcited by 0
  15. 2026LLMs as Post-hoc Auditors of Physiological Plausibility in Symbolic Regression: A Clinician-Evaluated Case Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMize: A Framework for Large Language Model-Based Numerical Optimization
  2. 2025Exploiting Edited Large Language Models as General Scientific Optimizers
  3. 2025Evolutionary thoughts: integration of large language models and evolutionary algorithms
  4. 2026OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling
  5. 2026FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization
  6. 2026Can Large Language Models Reason and Optimize Under Constraints?
  7. 2025Peering Inside the Black Box: Uncovering LLM Errors in Optimization Modelling through Component-Level Evaluation
  8. 2024An investigation on the use of Large Language Models for hyperparameter tuning in Evolutionary Algorithms