# LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms

arXiv: https://arxiv.org/abs/2505.21034 · HTML: https://mlsanity.com/paper/2505.21034/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 30 of 33)

- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model](https://mlsanity.com/paper/2401.02051/) (2024)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance](https://mlsanity.com/paper/2304.11127/) (2023)
- [High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces](https://mlsanity.com/paper/2103.00349/) (2021)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [Large Language Models to Enhance Bayesian Optimization](https://mlsanity.com/paper/2402.03921/) (2024)
- [Batch Bayesian Optimization via Local Penalization](https://mlsanity.com/paper/1505.08052/) (2015)
- [LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics](https://mlsanity.com/paper/2405.20132/) (2024)
- [Scalable Constrained Bayesian Optimization](https://mlsanity.com/paper/2002.08526/) (2020)
- [HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation](https://mlsanity.com/paper/2012.03826/) (2020)
- [πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization](https://mlsanity.com/paper/2204.11051/) (2022)
- [A Systematic Survey on Large Language Models for Algorithm Design](https://mlsanity.com/paper/2410.14716/) (2024)
- [Joint Entropy Search for Maximally-Informed Bayesian Optimization](https://mlsanity.com/paper/2206.04771/) (2022)
- [Algorithm Evolution Using Large Language Model](https://mlsanity.com/paper/2311.15249/) (2023)
- [Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces](https://mlsanity.com/paper/2304.11468/) (2023)
- [Greed is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisation](https://mlsanity.com/paper/1911.12809/) (2019)
- [A General Framework for Constrained Bayesian Optimization using Information-based Search](https://mlsanity.com/paper/1511.09422/) (2015)
- [ADO-LLM: Analog Design Bayesian Optimization with In-Context Learning of Large Language Models](https://mlsanity.com/paper/2406.18770/) (2024)
- [Are Random Decompositions all we need in High Dimensional Bayesian Optimisation?](https://mlsanity.com/paper/2301.12844/) (2023)
- [In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics](https://mlsanity.com/paper/2410.16309/) (2024)
- [IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics](https://mlsanity.com/paper/2111.04077/) (2021)
- [Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Models](https://mlsanity.com/paper/2407.10873/) (2024)
- [Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOB](https://mlsanity.com/paper/2303.00890/) (2023)
- [Asynchronous Batch Bayesian Optimisation with Improved Local Penalisation](https://mlsanity.com/paper/1901.10452/) (2019)
- [Evolve Cost-aware Acquisition Functions Using Large Language Models](https://mlsanity.com/paper/2404.16906/) (2024)
- [Differentiating the multipoint Expected Improvement for optimal batch design](https://mlsanity.com/paper/1503.05509/) (2015)

## Built on by (most cited first, 10 of 10)

- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priors](https://mlsanity.com/paper/2603.02792/) (2026)
- [Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization](https://mlsanity.com/paper/2605.20249/) (2026)
- [MALBO: Optimizing LLM-Based Multi-Agent Teams via Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2511.11788/) (2025)
- [CliffSearch: Structured Agentic Co-Evolution over Theory and Code for Scientific Algorithm Discovery](https://mlsanity.com/paper/2604.01210/) (2026)
- [An Open-Source Training Dataset for Foundation Models for Black-box Optimization](https://mlsanity.com/paper/2605.23417/) (2026)
- [AutoSG: LLM-Driven Solver Generation Solely from Task Prompts for Expensive Optimization](https://mlsanity.com/paper/2605.25658/) (2026)
- [LLM-Driven Evolutionary Generation of Multi-Objective Bayesian Optimization Algorithms](https://mlsanity.com/paper/2607.08791/) (2026)
- [Code-to-Harness: Distilling Black-Box Optimizers from Self-Play](https://mlsanity.com/paper/2609.09468/) (2026)
- [Hyper Algorithm Design Agent: Evolving Learnable Optimizer from Zero](https://mlsanity.com/paper/2609.35328/) (2026)

## Abstract (arXiv)

Bayesian optimization (BO) is a powerful class of algorithms for optimizing expensive black-box functions, but designing effective BO algorithms remains a manual, expertise-driven task. Recent advancements in Large Language Models (LLMs) have opened new avenues for automating scientific discovery, including the automatic design of optimization algorithms. While prior work has used LLMs within optimization loops or to generate non-BO algorithms, we tackle a new challenge: Using LLMs to automatically generate full BO algorithm code. Our framework uses an evolution strategy to guide an LLM in generating Python code that preserves the key components of BO algorithms: An initial design, a surrogate model, and an acquisition function. The LLM is prompted to produce multiple candidate algorithms, which are evaluated on the established Black-Box Optimization Benchmarking (BBOB) test suite from the COmparing Continuous Optimizers (COCO) platform. Based on their performance, top candidates are selected, combined, and mutated via controlled prompt variations, enabling iterative refinement. Despite no additional fine-tuning, the LLM-generated algorithms outperform state-of-the-art BO baselines in 19 (out of 24) BBOB functions in dimension 5 and generalize well to higher dimensions, and different tasks (from the Bayesmark framework). This work demonstrates that LLMs can serve as algorithmic co-designers, offering a new paradigm for automating BO development and accelerating the discovery of novel algorithmic combinations. The source code is provided at https://github.com/Ewendawi/LLaMEA-BO.