# When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets

arXiv: https://arxiv.org/abs/2607.16476 · HTML: https://mlsanity.com/paper/2607.16476/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 87 references that are in the map.
- First posted on arXiv 17 Jul 2026.

## Builds on (most cited first, 30 of 30)

- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [ParamILS: An Automatic Algorithm Configuration Framework](https://mlsanity.com/paper/1401.3492/) (2014)
- [YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization](https://mlsanity.com/paper/2109.03670/) (2021)
- [Finding Faster Configurations using FLASH](https://mlsanity.com/paper/1801.02175/) (2018)
- [Multi-Objectivizing Software Configuration Tuning (for a single performance concern)](https://mlsanity.com/paper/2106.01331/) (2021)
- [Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation](https://mlsanity.com/paper/2110.12654/) (2021)
- [Do Performance Aspirations Matter for Guiding Software Configuration Tuning?](https://mlsanity.com/paper/2301.03290/) (2023)
- [Transfer Learning for Performance Modeling of Configurable Systems: An Exploratory Analysis](https://mlsanity.com/paper/1709.02280/) (2017)
- [MMO: Meta Multi-Objectivization for Software Configuration Tuning](https://mlsanity.com/paper/2112.07303/) (2021)
- [An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing Systems](https://mlsanity.com/paper/1606.06543/) (2016)
- [Predicting Configuration Performance in Multiple Environments with Sequential Meta-learning](https://mlsanity.com/paper/2402.03183/) (2024)
- [Learning Software Configuration Spaces: A Systematic Literature Review](https://mlsanity.com/paper/1906.03018/) (2019)
- [Adapting Multi-objectivized Software Configuration Tuning](https://mlsanity.com/paper/2404.04744/) (2024)
- [Planning Landscape Analysis for Self-Adaptive Systems](https://mlsanity.com/paper/2203.12472/) (2022)
- [Distilled Lifelong Self-Adaptation for Configurable Systems](https://mlsanity.com/paper/2501.00840/) (2025)
- [Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning](https://mlsanity.com/paper/2501.01876/) (2025)
- ["Sampling"' as a Baseline Optimizer for Search-based Software Engineering](https://mlsanity.com/paper/1608.07617/) (2016)
- [An Evolutionary Study of Configuration Design and Implementation in Cloud Systems](https://mlsanity.com/paper/2102.07052/) (2021)
- [Whence to Learn? Transferring Knowledge in Configurable Systems using BEETLE](https://mlsanity.com/paper/1911.01817/) (2019)
- [Is Large Language Model Good at Database Knob Tuning? A Comprehensive Experimental Evaluation](https://mlsanity.com/paper/2408.02213/) (2024)
- [CM-CASL: Comparison-based Performance Modeling of Software Systems via Collaborative Active and Semisupervised Learning](https://mlsanity.com/paper/2303.15967/) (2023)
- [CAMEO: A Causal Transfer Learning Approach for Performance Optimization of Configurable Computer Systems](https://mlsanity.com/paper/2306.07888/) (2023)
- [Learning from Very Little Data: On the Value of Landscape Analysis for Predicting Software Project Health](https://mlsanity.com/paper/2301.06577/) (2023)
- [Rethinking Performance Analysis for Configurable Software Systems: A Case Study from a Fitness Landscape Perspective](https://mlsanity.com/paper/2412.16888/) (2024)
- [Can Large Language Models Improve SE Active Learning via Warm-Starts?](https://mlsanity.com/paper/2501.00125/) (2024)
- [BINGO! Simple Optimizers Win Big if Problems Collapse to a Few Buckets](https://mlsanity.com/paper/2506.04509/) (2025)
- [Do We Really Need to Approach the Entire Pareto Front in Many-Objective Bayesian Optimisation?](https://mlsanity.com/paper/2604.09417/) (2026)
- [EMIT: Micro-Invasive Database Configuration Tuning](https://mlsanity.com/paper/2406.00616/) (2024)

## Built on by (most cited first, 1 of 1)

- [Converge Then Diversify: Decoupling Convergence and Diversity in Multi-Objective Bayesian Optimisation](https://mlsanity.com/paper/2609.13396/) (2026)

## Abstract (arXiv)

Software configuration tuning is crucial for optimising system performance, and various optimisers have emerged over the last decade. Yet, the time required during the tuning process may vary across systems. In some systems (e.g., PostgreSQL), it may take a few minutes to measure a configuration, whereas in some others (e.g., MariaDB), it can take several hours. Moreover, even within the same system, users may have varying budgets and preferred settings. This naturally raises a question -- Given a budget level, which optimiser is the best choice for SE practitioners? This matters because optimisers usually have their own ``comfort zone'' and may perform very differently under distinct budgets. In this paper, we aim to answer this question. We systematically evaluate eight well-established optimisers across 22 configurable systems under varying budget levels. We find that, unsurprisingly, model-based optimisers (e.g., SMAC) are well-suited under tight budgets, and model-free optimisers (e.g., GAs) become superior with more generous budgets. However, interestingly, there is one optimiser, FLASH, that performs consistently well on most systems regardless of budgets. We lastly investigate the reasons behind this phenomenon and find that many systems possess good local optima (with large basins of attraction), allowing greedy optimisers (e.g., FLASH) to achieve strong performance. Source code, data, and supplementary materials of this work are available at https://anonymous.4open.science/r/Config-W2W-98B2.