# Tuning the Tuner: Introducing Hyperparameter Optimization for Auto-Tuning

arXiv: https://arxiv.org/abs/2509.26300 · HTML: https://mlsanity.com/paper/2509.26300/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 44 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Builds on (most cited first, 17 of 17)

- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [A Survey on Compiler Autotuning using Machine Learning](https://mlsanity.com/paper/1801.04405/) (2018)
- [CLTune: A Generic Auto-Tuner for OpenCL Kernels](https://mlsanity.com/paper/1703.06503/) (2017)
- [ytopt: Autotuning Scientific Applications for Energy Efficiency at Large Scales](https://mlsanity.com/paper/2303.16245/) (2023)
- [BaCO: A Fast and Portable Bayesian Compiler Optimization Framework](https://mlsanity.com/paper/2212.11142/) (2022)
- [CLBlast: A Tuned OpenCL BLAS Library](https://mlsanity.com/paper/1705.05249/) (2017)
- [Machine Learning Based Auto-tuning for Enhanced OpenCL Performance Portability](https://mlsanity.com/paper/1506.00842/) (2015)
- [A Benchmark Set of Highly-efficient CUDA and OpenCL Kernels and its Dynamic Autotuning with Kernel Tuning Toolkit](https://mlsanity.com/paper/1910.08498/) (2019)
- [Going green: optimizing GPUs for energy efficiency through model-steered auto-tuning](https://mlsanity.com/paper/2211.07260/) (2022)
- [Kernel Launcher: C++ Library for Optimal-Performance Portable CUDA Applications](https://mlsanity.com/paper/2303.12374/) (2023)
- [Bringing Auto-tuning to HIP: Analysis of Tuning Impact and Difficulty on AMD and Nvidia GPUs](https://mlsanity.com/paper/2407.11488/) (2024)
- [Optimizing performance per watt on GPUs in High Performance Computing: temperature, frequency and voltage effects](https://mlsanity.com/paper/1407.8116/) (2014)
- [Auto-Tuning Dedispersion for Many-Core Accelerators](https://mlsanity.com/paper/1601.05052/) (2016)
- [Lessons learned in a decade of research software engineering GPU applications](https://mlsanity.com/paper/2005.13227/) (2020)
- [Benchmarking optimization algorithms for auto-tuning GPU kernels](https://mlsanity.com/paper/2210.01465/) (2022)
- [Cross-Platform Performance Portability Using Highly Parametrized SYCL Kernels](https://mlsanity.com/paper/1904.05347/) (2019)
- [The Tensor-Core Beamformer: A High-Speed Signal-Processing Library for Multidisciplinary Use](https://mlsanity.com/paper/2505.03269/) (2025)

## Built on by (most cited first, 1 of 1)

- [Automated Algorithm Design for Auto-Tuning Optimizers](https://mlsanity.com/paper/2510.17899/) (2025)

## Abstract (arXiv)

Automatic performance tuning (auto-tuning) is widely used to optimize performance-critical applications across many scientific domains by finding the best program variant among many choices. Efficient optimization algorithms are crucial for navigating the vast and complex search spaces in auto-tuning. As is well known in the context of machine learning and similar fields, hyperparameters critically shape optimization algorithm efficiency. Yet for auto-tuning frameworks, these hyperparameters are almost never tuned, and their potential performance impact has not been studied. We present a novel method for general hyperparameter tuning of optimization algorithms for auto-tuning, thus "tuning the tuner". In particular, we propose a robust statistical method for evaluating hyperparameter performance across search spaces, publish a FAIR data set and software for reproducibility, and present a simulation mode that replays previously recorded tuning data, lowering the costs of hyperparameter tuning by two orders of magnitude. We show that even limited hyperparameter tuning can improve auto-tuner performance by 94.8% on average, and establish that the hyperparameters themselves can be optimized efficiently with meta-strategies (with an average improvement of 204.7%), demonstrating the often overlooked hyperparameter tuning as a powerful technique for advancing auto-tuning research and practice.