# Automated Algorithm Design for Auto-Tuning Optimizers

arXiv: https://arxiv.org/abs/2510.17899 · HTML: https://mlsanity.com/paper/2510.17899/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 55 references that are in the map.
- First posted on arXiv 19 Oct 2025.

## Builds on (most cited first, 21 of 21)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution](https://mlsanity.com/paper/2402.01145/) (2024)
- [Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model](https://mlsanity.com/paper/2401.02051/) (2024)
- [LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics](https://mlsanity.com/paper/2405.20132/) (2024)
- [In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics](https://mlsanity.com/paper/2410.16309/) (2024)
- [CLTune: A Generic Auto-Tuner for OpenCL Kernels](https://mlsanity.com/paper/1703.06503/) (2017)
- [Autotuning PolyBench Benchmarks with LLVM Clang/Polly Loop Optimization Pragmas Using Bayesian Optimization](https://mlsanity.com/paper/2010.08040/) (2020)
- [ytopt: Autotuning Scientific Applications for Energy Efficiency at Large Scales](https://mlsanity.com/paper/2303.16245/) (2023)
- [BaCO: A Fast and Portable Bayesian Compiler Optimization Framework](https://mlsanity.com/paper/2212.11142/) (2022)
- [BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics](https://mlsanity.com/paper/2504.20183/) (2025)
- [Machine Learning Based Auto-tuning for Enhanced OpenCL Performance Portability](https://mlsanity.com/paper/1506.00842/) (2015)
- [A Benchmark Set of Highly-efficient CUDA and OpenCL Kernels and its Dynamic Autotuning with Kernel Tuning Toolkit](https://mlsanity.com/paper/1910.08498/) (2019)
- [Bringing Auto-tuning to HIP: Analysis of Tuning Impact and Difficulty on AMD and Nvidia GPUs](https://mlsanity.com/paper/2407.11488/) (2024)
- [Auto-Tuning Dedispersion for Many-Core Accelerators](https://mlsanity.com/paper/1601.05052/) (2016)
- [Lessons learned in a decade of research software engineering GPU applications](https://mlsanity.com/paper/2005.13227/) (2020)
- [Benchmarking optimization algorithms for auto-tuning GPU kernels](https://mlsanity.com/paper/2210.01465/) (2022)
- [Towards a Benchmarking Suite for Kernel Tuners](https://mlsanity.com/paper/2303.08976/) (2023)
- [Using hardware performance counters to speed up autotuning convergence on GPUs](https://mlsanity.com/paper/2102.05297/) (2021)
- [Efficient Construction of Large Search Spaces for Auto-Tuning](https://mlsanity.com/paper/2509.26253/) (2025)
- [Tuning the Tuner: Introducing Hyperparameter Optimization for Auto-Tuning](https://mlsanity.com/paper/2509.26300/) (2025)

## Abstract (arXiv)

Automatic performance tuning (auto-tuning) is essential for optimizing high-performance applications, where vast and irregular search spaces make manual exploration infeasible. While auto-tuners traditionally rely on classical approaches such as evolutionary, annealing, or surrogate-based optimizers, designing algorithms that efficiently find near-optimal configurations robustly across diverse tasks is challenging. We propose a new paradigm: using large language models (LLMs) to automatically generate optimization algorithms tailored to auto-tuning problems. We introduce a framework that prompts LLMs with problem descriptions and search space characteristics to synthesize, test, and iteratively refine specialized optimizers. These generated algorithms are evaluated on four real-world auto-tuning applications across six hardware platforms and compared against the state-of-the-art in two contemporary auto-tuning frameworks. The evaluation demonstrates that providing additional application- and search space-specific information in the generation stage results in an average performance improvement of 30.7% and 14.6%, respectively. In addition, our results show that LLM-generated optimizers can rival, and in various cases outperform, existing human-designed algorithms, with our best-performing generated optimization algorithms achieving an average 72.4% improvement over state-of-the-art optimizers for auto-tuning.