# Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery

arXiv: https://arxiv.org/abs/2507.03605 · HTML: https://mlsanity.com/paper/2507.03605/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 4 Jul 2025.

## Builds on (most cited first, 20 of 20)

- [ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution](https://mlsanity.com/paper/2402.01145/) (2024)
- [Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model](https://mlsanity.com/paper/2401.02051/) (2024)
- [Multi-Step Reasoning with Large Language Models, a Survey](https://mlsanity.com/paper/2407.11511/) (2024)
- [LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics](https://mlsanity.com/paper/2405.20132/) (2024)
- [A Systematic Survey on Large Language Models for Algorithm Design](https://mlsanity.com/paper/2410.14716/) (2024)
- [In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics](https://mlsanity.com/paper/2410.16309/) (2024)
- [IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics](https://mlsanity.com/paper/2111.04077/) (2021)
- [Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Models](https://mlsanity.com/paper/2407.10873/) (2024)
- [Tuning as a Means of Assessing the Benefits of New Ideas in Interplay with Existing Algorithmic Modules](https://mlsanity.com/paper/2102.12905/) (2021)
- [Local Optima Networks: A New Model of Combinatorial Fitness Landscapes](https://mlsanity.com/paper/1402.2959/) (2014)
- [Evolving the Structure of Evolution Strategies](https://mlsanity.com/paper/1610.05231/) (2016)
- [Modular Differential Evolution](https://mlsanity.com/paper/2304.09524/) (2023)
- [Code Evolution Graphs: Understanding Large Language Model Driven Design of Algorithms](https://mlsanity.com/paper/2503.16668/) (2025)
- [BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics](https://mlsanity.com/paper/2504.20183/) (2025)
- [Controlling the Mutation in Large Language Models for the Efficient Evolution of Algorithms](https://mlsanity.com/paper/2412.03250/) (2024)
- [Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systems](https://mlsanity.com/paper/2505.15741/) (2025)
- [Average Convergence Rate of Evolutionary Algorithms](https://mlsanity.com/paper/1504.08117/) (2015)
- [Phenotype Search Trajectory Networks for Linear Genetic Programming](https://mlsanity.com/paper/2211.08516/) (2022)
- [Stalling in Space: Attractor Analysis for any Algorithm](https://mlsanity.com/paper/2412.15848/) (2024)
- [Building Trust in Black-box Optimization: A Comprehensive Framework for Explainability](https://mlsanity.com/paper/2410.14573/) (2024)

## Built on by (most cited first, 4 of 4)

- [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025)
- [What Makes an LLM a Good Optimizer? A Trajectory Analysis of LLM-Guided Evolutionary Search](https://mlsanity.com/paper/2604.19440/) (2026)
- [LLM Driven Design of Continuous Optimization Problems with Controllable High-level Properties](https://mlsanity.com/paper/2601.18846/) (2026)
- [Behavior and Representation in Open-Weight Large Language Models for Combinatorial Optimization: From Feature Extraction to Algorithm Selection](https://mlsanity.com/paper/2512.13374/) (2025)

## Abstract (arXiv)

We investigate the behaviour space of meta-heuristic optimisation algorithms automatically generated by Large Language Model driven algorithm discovery methods. Using the Large Language Evolutionary Algorithm (LLaMEA) framework with a GPT o4-mini LLM, we iteratively evolve black-box optimisation heuristics, evaluated on 10 functions from the BBOB benchmark suite. Six LLaMEA variants, featuring different mutation prompt strategies, are compared and analysed. We log dynamic behavioural metrics including exploration, exploitation, convergence and stagnation measures, for each run, and analyse these via visual projections and network-based representations. Our analysis combines behaviour-based projections, Code Evolution Graphs built from static code features, performance convergence curves, and behaviour-based Search Trajectory Networks. The results reveal clear differences in search dynamics and algorithm structures across LLaMEA configurations. Notably, the variant that employs both a code simplification prompt and a random perturbation prompt in a 1+1 elitist evolution strategy, achieved the best performance, with the highest Area Over the Convergence Curve. Behaviour-space visualisations show that higher-performing algorithms exhibit more intensive exploitation behaviour and faster convergence with less stagnation. Our findings demonstrate how behaviour-space analysis can explain why certain LLM-designed heuristics outperform others and how LLM-driven algorithm discovery navigates the open-ended and complex search space of algorithms. These findings provide insights to guide the future design of adaptive LLM-driven algorithm generators.