# Revisiting Bayesian Optimization in the light of the COCO benchmark

arXiv: https://arxiv.org/abs/2103.16649 · HTML: https://mlsanity.com/paper/2103.16649/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 70 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 18 of 18)

- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [Bayesian Optimization in a Billion Dimensions via Random Embeddings](https://mlsanity.com/paper/1301.1942/) (2013)
- [High Dimensional Bayesian Optimisation and Bandits via Additive Models](https://mlsanity.com/paper/1503.01673/) (2015)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [Input Warping for Bayesian Optimization of Non-stationary Functions](https://mlsanity.com/paper/1402.0929/) (2014)
- [BOCK : Bayesian Optimization with Cylindrical Kernels](https://mlsanity.com/paper/1806.01619/) (2018)
- [Greed is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisation](https://mlsanity.com/paper/1911.12809/) (2019)
- [BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization](https://mlsanity.com/paper/1910.06403/) (2019)
- [TREGO: a Trust-Region Framework for Efficient Global Optimization](https://mlsanity.com/paper/2101.06808/) (2021)
- [A Bayesian approach to constrained single- and multi-objective optimization](https://mlsanity.com/paper/1510.00503/) (2015)
- [Optimization, fast and slow: optimally switching between local and Bayesian optimization](https://mlsanity.com/paper/1805.08610/) (2018)
- [BayesOpt: A Bayesian Optimization Library for Nonlinear Optimization, Experimental Design and Bandits](https://mlsanity.com/paper/1405.7430/) (2014)
- [GPflowOpt: A Bayesian Optimization Library using TensorFlow](https://mlsanity.com/paper/1711.03845/) (2017)
- [Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters](https://mlsanity.com/paper/1908.06674/) (2019)
- [Global sensitivity analysis for optimization with variable selection](https://mlsanity.com/paper/1811.04646/) (2018)
- [Modeling and Optimization with Gaussian Processes in Reduced Eigenbases -- Extended Version](https://mlsanity.com/paper/1908.11272/) (2019)
- [Expected Improvement versus Predicted Value in Surrogate-Based Optimization](https://mlsanity.com/paper/2001.02957/) (2020)

## Built on by (most cited first, 10 of 10)

- [HPO X ELA: Investigating Hyperparameter Optimization Landscapes by Means of Exploratory Landscape Analysis](https://mlsanity.com/paper/2208.00220/) (2022)
- [High-dimensional mixed-categorical Gaussian processes with application to multidisciplinary design optimization for a green aircraft](https://mlsanity.com/paper/2311.06130/) (2023)
- [Bayesian optimization for mixed variables using an adaptive dimension reduction process: applications to aircraft design](https://mlsanity.com/paper/2504.08682/) (2025)
- [A comparison of mixed-variables Bayesian optimization approaches](https://mlsanity.com/paper/2111.01533/) (2021)
- [MONGOOSE: Path-wise Smooth Bayesian Optimisation via Meta-learning](https://mlsanity.com/paper/2302.11533/) (2023)
- [Modeling Hierarchical Spaces: A Review and Unified Framework for Surrogate-Based Architecture Design](https://mlsanity.com/paper/2506.22621/) (2025)
- [Co-Learning Bayesian Optimization](https://mlsanity.com/paper/2501.13332/) (2025)
- [Bayesian Quality-Diversity approaches for constrained optimization problems with mixed continuous, discrete and categorical variables](https://mlsanity.com/paper/2310.05955/) (2023)
- [mlr3mbo: Bayesian Optimization in R](https://mlsanity.com/paper/2603.29730/) (2026)
- [tidyHEBO: Improving the Robustness of Heteroskedastic Evolutionary Bayesian Optimization](https://mlsanity.com/paper/2607.10669/) (2026)

## Abstract (arXiv)

It is commonly believed that Bayesian optimization (BO) algorithms are highly efficient for optimizing numerically costly functions. However, BO is not often compared to widely different alternatives, and is mostly tested on narrow sets of problems (multimodal, low-dimensional functions), which makes it difficult to assess where (or if) they actually achieve state-of-the-art performance. Moreover, several aspects in the design of these algorithms vary across implementations without a clear recommendation emerging from current practices, and many of these design choices are not substantiated by authoritative test campaigns. This article reports a large investigation about the effects on the performance of (Gaussian process based) BO of common and less common design choices. The experiments are carried out with the established COCO (COmparing Continuous Optimizers) software. It is found that a small initial budget, a quadratic trend, high-quality optimization of the acquisition criterion bring consistent progress. Using the GP mean as an occasional acquisition contributes to a negligible additional improvement. Warping degrades performance. The Mat'ern 5/2 kernel is a good default but it may be surpassed by the exponential kernel on irregular functions. Overall, the best EGO variants are competitive or improve over state-of-the-art algorithms in dimensions less or equal to 5 for multimodal functions. The code developed for this study makes the new version (v2.1.1) of the R package DiceOptim available on CRAN. The structure of the experiments by function groups allows to define priorities for future research on Bayesian optimization.