# BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics

arXiv: https://arxiv.org/abs/2504.20183 · HTML: https://mlsanity.com/paper/2504.20183/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Builds on (most cited first, 17 of 17)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics](https://mlsanity.com/paper/2405.20132/) (2024)
- [A Systematic Survey on Large Language Models for Algorithm Design](https://mlsanity.com/paper/2410.14716/) (2024)
- [Benchmarking in Optimization: Best Practice and Open Issues](https://mlsanity.com/paper/2007.03488/) (2020)
- [In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics](https://mlsanity.com/paper/2410.16309/) (2024)
- [IOHanalyzer: Detailed Performance Analyses for Iterative Optimization Heuristics](https://mlsanity.com/paper/2007.03953/) (2020)
- [Tuning as a Means of Assessing the Benefits of New Ideas in Interplay with Existing Algorithmic Modules](https://mlsanity.com/paper/2102.12905/) (2021)
- [Reproducibility in Evolutionary Computation](https://mlsanity.com/paper/2102.03380/) (2021)
- [MA-BBOB: A Problem Generator for Black-Box Optimization Using Affine Combinations and Shifts](https://mlsanity.com/paper/2312.11083/) (2023)
- [Code Evolution Graphs: Understanding Large Language Model Driven Design of Algorithms](https://mlsanity.com/paper/2503.16668/) (2025)
- [BBOB Instance Analysis: Landscape Properties and Algorithm Performance across Problem Instances](https://mlsanity.com/paper/2211.16318/) (2022)
- [Explainable Benchmarking for Iterative Optimization Heuristics](https://mlsanity.com/paper/2401.17842/) (2024)
- [Controlling the Mutation in Large Language Models for the Efficient Evolution of Algorithms](https://mlsanity.com/paper/2412.03250/) (2024)
- [Generalization Ability of Feature-based Performance Prediction Models: A Statistical Analysis across Benchmarks](https://mlsanity.com/paper/2405.12259/) (2024)
- [Optimizing Photonic Structures with Large Language Model Driven Algorithm Discovery](https://mlsanity.com/paper/2503.19742/) (2025)
- [Analysis of modular CMA-ES on strict box-constrained problems in the SBOX-COST benchmarking suite](https://mlsanity.com/paper/2305.15102/) (2023)

## Built on by (most cited first, 9 of 9)

- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery](https://mlsanity.com/paper/2507.03605/) (2025)
- [Solve it with EASE](https://mlsanity.com/paper/2509.18108/) (2025)
- [Overlooked Safety Vulnerability in LLMs: Malicious Intelligent Optimization Algorithm Request and its Jailbreak](https://mlsanity.com/paper/2601.00213/) (2026)
- [LLaMEA-SAGE: Guiding Automated Algorithm Design with Structural Feedback from Explainable AI](https://mlsanity.com/paper/2601.21511/) (2026)
- [Automated Algorithm Design for Auto-Tuning Optimizers](https://mlsanity.com/paper/2510.17899/) (2025)
- [From Performance to Understanding: A Vision for Explainable Automated Algorithm Design](https://mlsanity.com/paper/2511.16201/) (2025)
- [LLM-Driven Evolutionary Generation of Multi-Objective Bayesian Optimization Algorithms](https://mlsanity.com/paper/2607.08791/) (2026)
- [Evolving Executable Pipeline Programs for AutoML with Language Models](https://mlsanity.com/paper/2608.16416/) (2026)

## Abstract (arXiv)

The application of Large Language Models (LLMs) for Automated Algorithm Discovery (AAD), particularly for optimisation heuristics, is an emerging field of research. This emergence necessitates robust, standardised benchmarking practices to rigorously evaluate the capabilities and limitations of LLM-driven AAD methods and the resulting generated algorithms, especially given the opacity of their design process and known issues with existing benchmarks. To address this need, we introduce BLADE (Benchmark suite for LLM-driven Automated Design and Evolution), a modular and extensible framework specifically designed for benchmarking LLM-driven AAD methods in a continuous black-box optimisation context. BLADE integrates collections of benchmark problems (including MA-BBOB and SBOX-COST among others) with instance generators and textual descriptions aimed at capability-focused testing, such as generalisation, specialisation and information exploitation. It offers flexible experimental setup options, standardised logging for reproducibility and fair comparison, incorporates methods for analysing the AAD process (e.g., Code Evolution Graphs and various visualisation approaches) and facilitates comparison against human-designed baselines through integration with established tools like IOHanalyser and IOHexplainer. BLADE provides an `out-of-the-box' solution to systematically evaluate LLM-driven AAD approaches. The framework is demonstrated through two distinct use cases exploring mutation prompt strategies and function specialisation.