# An Evolutionary Framework for Automatic Optimization Benchmark Generation via Large Language Models

arXiv: https://arxiv.org/abs/2601.12723 · HTML: https://mlsanity.com/paper/2601.12723/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 42 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [Large Language Models as Evolutionary Optimizers](https://mlsanity.com/paper/2310.19046/) (2023)
- [Language Model Crossover: Variation through Few-Shot Prompting](https://mlsanity.com/paper/2302.12170/) (2023)
- [LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics](https://mlsanity.com/paper/2405.20132/) (2024)
- [An Easy-to-use Real-world Multi-objective Optimization Problem Suite](https://mlsanity.com/paper/2009.12867/) (2020)
- [Large Language Models As Evolution Strategies](https://mlsanity.com/paper/2402.18381/) (2024)
- [Benchmarking Evolutionary Algorithms For Single Objective Real-valued Constrained Optimization - A Critical Review](https://mlsanity.com/paper/1806.04563/) (2018)
- [Evolving test instances of the Hamiltonian completion problem](https://mlsanity.com/paper/2011.02291/) (2020)
- [Evolving Benchmark Functions to Compare Evolutionary Algorithms via Genetic Programming](https://mlsanity.com/paper/2403.14146/) (2024)

## Abstract (arXiv)

Optimization benchmarks play a fundamental role in assessing algorithm performance; however, existing artificial benchmarks often fail to capture the diversity and irregularity of real-world problem structures, while benchmarks derived from real-world problems are costly and difficult to construct. To address these challenges, we propose an evolutionary automatic benchmark generation framework that leverages a large language model (LLM) as a generative operator, termed the LLM-driven evolutionary benchmark generator (LLM-EBG). In this framework, the LLM serves as an evolutionary operator that generates and evolves benchmark problems within a flexible, expressive representation space. As a case study, we generate unconstrained single-objective continuous minimization problems represented as mathematical expressions designed to induce significant performance differences between a genetic algorithm (GA) and differential evolution (DE). Experimental results show that LLM-EBG successfully produces benchmark problems in which the designated target algorithm consistently outperforms the comparative algorithm in more than 80% of trials. Furthermore, exploratory landscape analysis reveals that benchmarks favoring GA are highly sensitive to variable scaling, demonstrating that the proposed framework can generate problems with distinct geometric characteristics that reflect the intrinsic search behaviors of different optimization algorithms.