# In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics arXiv: https://arxiv.org/abs/2410.16309 ยท HTML: https://mlsanity.com/paper/2410.16309/ ## Key facts - Cited by 28 papers in the map (data through 14 Oct 2026). - Builds on 9 of its 34 references that are in the map. - First posted on arXiv 7 Oct 2024. ## Builds on (most cited first, 9 of 9) - [Attention, Learn to Solve Routing Problems!](https://mlsanity.com/paper/1803.08475/) (2018) - [An Efficient Graph Convolutional Network Technique for the Travelling Salesman Problem](https://mlsanity.com/paper/1906.01227/) (2019) - [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021) - [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021) - [Graph Neural Network Guided Local Search for the Traveling Salesperson Problem](https://mlsanity.com/paper/2110.05291/) (2021) - [BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters](https://mlsanity.com/paper/1908.06756/) (2019) - [IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics](https://mlsanity.com/paper/2111.04077/) (2021) - [EB-GLS: An Improved Guided Local Search Based on the Big Valley Structure](https://mlsanity.com/paper/1709.07576/) (2017) - [Using the Empirical Attainment Function for Analyzing Single-objective Black-box Optimization Algorithms](https://mlsanity.com/paper/2404.02031/) (2024) ## Built on by (most cited first, 28 of 28) - [A Systematic Survey on Large Language Models for Algorithm Design](https://mlsanity.com/paper/2410.14716/) (2024) - [Language-Based Bayesian Optimization Research Assistant (BORA)](https://mlsanity.com/paper/2501.16224/) (2025) - [Code Evolution Graphs: Understanding Large Language Model Driven Design of Algorithms](https://mlsanity.com/paper/2503.16668/) (2025) - [A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solving](https://mlsanity.com/paper/2509.08269/) (2025) - [LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms](https://mlsanity.com/paper/2505.21034/) (2025) - [Fitness Landscape of Large Language Model-Assisted Automated Algorithm Search](https://mlsanity.com/paper/2504.19636/) (2025) - [BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics](https://mlsanity.com/paper/2504.20183/) (2025) - [OptiChat: Bridging Optimization Models and Practitioners with Large Language Models](https://mlsanity.com/paper/2501.08406/) (2025) - [Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systems](https://mlsanity.com/paper/2505.15741/) (2025) - [EvoCAD: Evolutionary CAD Code Generation with Vision Language Models](https://mlsanity.com/paper/2510.11631/) (2025) - [Autonomous Radiotherapy Treatment Planning Using DOLA: A Privacy-Preserving, LLM-Based Optimization Agent](https://mlsanity.com/paper/2503.17553/) (2025) - [Behaviour Space Analysis of LLM-driven Meta-heuristic Discovery](https://mlsanity.com/paper/2507.03605/) (2025) - [From Understanding to Excelling: Template-Free Algorithm Design through Structural-Functional Co-Evolution](https://mlsanity.com/paper/2503.10721/) (2025) - [Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH](https://mlsanity.com/paper/2601.12355/) (2026) - [LLM Driven Design of Continuous Optimization Problems with Controllable High-level Properties](https://mlsanity.com/paper/2601.18846/) (2026) - [From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priors](https://mlsanity.com/paper/2603.02792/) (2026) - [Solve it with EASE](https://mlsanity.com/paper/2509.18108/) (2025) - [irace-evo: Automatic Algorithm Configuration Extended With LLM-Based Code Evolution](https://mlsanity.com/paper/2511.14794/) (2025) - [LLaMEA-SAGE: Guiding Automated Algorithm Design with Structural Feedback from Explainable AI](https://mlsanity.com/paper/2601.21511/) (2026) - [BEAM: Bi-level Memory-adaptive Algorithmic Evolution for LLM-Powered Heuristic Design](https://mlsanity.com/paper/2604.12898/) (2026) - [SpecAHD: Localize to Specialize for Automated Heuristic Design in Large-Scale Routing Problems](https://mlsanity.com/paper/2607.23676/) (2026) - [ATLAS: Scaffold-Free Algorithm Synthesis by LLMs via Embedding-Guided Quality-Diversity Search](https://mlsanity.com/paper/2608.15546/) (2026) - [Automated Algorithm Design for Auto-Tuning Optimizers](https://mlsanity.com/paper/2510.17899/) (2025) - [From Performance to Understanding: A Vision for Explainable Automated Algorithm Design](https://mlsanity.com/paper/2511.16201/) (2025) - [Stability as a Liability:Systematic Breakdown of Linguistic Structure in LLMs](https://mlsanity.com/paper/2601.18588/) (2026) - [LLM-Driven Evolutionary Generation of Multi-Objective Bayesian Optimization Algorithms](https://mlsanity.com/paper/2607.08791/) (2026) - [Evolving Executable Pipeline Programs for AutoML with Language Models](https://mlsanity.com/paper/2608.16416/) (2026) - [PINNForge: Execution-Grounded Evolutionary Design of Physics-Informed Neural Networks](https://mlsanity.com/paper/2609.23023/) (2026) ## Abstract (arXiv) Large Language Models (LLMs) have shown great potential in automatically generating and optimizing (meta)heuristics, making them valuable tools in heuristic optimization tasks. However, LLMs are generally inefficient when it comes to fine-tuning hyper-parameters of the generated algorithms, often requiring excessive queries that lead to high computational and financial costs. This paper presents a novel hybrid approach, LLaMEA-HPO, which integrates the open source LLaMEA (Large Language Model Evolutionary Algorithm) framework with a Hyper-Parameter Optimization (HPO) procedure in the loop. By offloading hyper-parameter tuning to an HPO procedure, the LLaMEA-HPO framework allows the LLM to focus on generating novel algorithmic structures, reducing the number of required LLM queries and improving the overall efficiency of the optimization process. We empirically validate the proposed hybrid framework on benchmark problems, including Online Bin Packing, Black-Box Optimization, and the Traveling Salesperson Problem. Our results demonstrate that LLaMEA-HPO achieves superior or comparable performance compared to existing LLM-driven frameworks while significantly reducing computational costs. This work highlights the importance of separating algorithmic innovation and structural code search from parameter tuning in LLM-driven code optimization and offers a scalable approach to improve the efficiency and effectiveness of LLM-based code generation.