# EXPObench: Benchmarking Surrogate-based Optimisation Algorithms on Expensive Black-box Functions

arXiv: https://arxiv.org/abs/2106.04618 · HTML: https://mlsanity.com/paper/2106.04618/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 70 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [Combinatorial Bayesian Optimization using the Graph Cartesian Product](https://mlsanity.com/paper/1902.00448/) (2019)
- [Bayesian Optimisation over Multiple Continuous and Categorical Inputs](https://mlsanity.com/paper/1906.08878/) (2019)
- [Benchmarking the Performance of Bayesian Optimization across Multiple Experimental Materials Science Domains](https://mlsanity.com/paper/2106.01309/) (2021)
- [IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics](https://mlsanity.com/paper/1810.05281/) (2018)
- [DACBench: A Benchmark Library for Dynamic Algorithm Configuration](https://mlsanity.com/paper/2105.08541/) (2021)
- [Safe Contextual Bayesian Optimization for Sustainable Room Temperature PID Control Tuning](https://mlsanity.com/paper/1906.12086/) (2019)
- [Black-box Mixed-Variable Optimisation using a Surrogate Model that Satisfies Integer Constraints](https://mlsanity.com/paper/2006.04508/) (2020)
- [Continuous surrogate-based optimization algorithms are well-suited for expensive discrete problems](https://mlsanity.com/paper/2011.03431/) (2020)
- [pySOT and POAP: An event-driven asynchronous framework for surrogate optimization](https://mlsanity.com/paper/1908.00420/) (2019)
- [Generative Melody Composition with Human-in-the-Loop Bayesian Optimization](https://mlsanity.com/paper/2010.03190/) (2020)
- [Black-box Combinatorial Optimization using Models with Integer-valued Minima](https://mlsanity.com/paper/1911.08817/) (2019)
- [Identifying Properties of Real-World Optimisation Problems through a Questionnaire](https://mlsanity.com/paper/2011.05547/) (2020)
- [Online Optimization with Costly and Noisy Measurements using Random Fourier Expansions](https://mlsanity.com/paper/1603.09620/) (2016)
- [Machine learning for improving performance in an evolutionary algorithm for minimum path with uncertain costs given by massively simulated scenarios](https://mlsanity.com/paper/2102.01830/) (2021)

## Built on by (most cited first, 9 of 9)

- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithms](https://mlsanity.com/paper/2410.20848/) (2024)
- [Symmetric Replay Training: Enhancing Sample Efficiency in Deep Reinforcement Learning for Combinatorial Optimization](https://mlsanity.com/paper/2306.01276/) (2023)
- [Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym](https://mlsanity.com/paper/2310.00077/) (2023)
- [Multi-fidelity Gaussian process surrogate modeling for regression problems in physics](https://mlsanity.com/paper/2404.11965/) (2024)
- [Bayesian Optimisation Against Climate Change: Applications and Benchmarks](https://mlsanity.com/paper/2306.04343/) (2023)
- [Performance Comparison of Surrogate-Assisted Evolutionary Algorithms on Computational Fluid Dynamics Problems](https://mlsanity.com/paper/2402.16455/) (2024)
- [BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks](https://mlsanity.com/paper/2605.17000/) (2026)
- [Optimal Transport-based Permutation-Invariant Bayesian Optimization of Offshore Wind Farm Layouts](https://mlsanity.com/paper/2606.00009/) (2026)

## Abstract (arXiv)

Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions and to give substantial advice on which method to use when. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different real-life applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. We also provide rules of thumb for which surrogate algorithm to use in which situation. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the performance of the six algorithms on all evaluated problem instances. This results in a unique new dataset that lowers the bar for researching new methods as the number of expensive evaluations required for comparison is significantly reduced.