# Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters

arXiv: https://arxiv.org/abs/1908.06674 · HTML: https://mlsanity.com/paper/1908.06674/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2019.

## Built on by (most cited first, 16 of 16)

- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [Using Large Language Models for Hyperparameter Optimization](https://mlsanity.com/paper/2312.04528/) (2023)
- [YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization](https://mlsanity.com/paper/2109.03670/) (2021)
- [Position: A Call to Action for a Human-Centered AutoML Paradigm](https://mlsanity.com/paper/2406.03348/) (2024)
- [Revisiting Bayesian Optimization in the light of the COCO benchmark](https://mlsanity.com/paper/2103.16649/) (2021)
- [PI is back! Switching Acquisition Functions in Bayesian Optimization](https://mlsanity.com/paper/2211.01455/) (2022)
- [Self-Adjusting Weighted Expected Improvement for Bayesian Optimization](https://mlsanity.com/paper/2306.04262/) (2023)
- [Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers](https://mlsanity.com/paper/2111.14756/) (2021)
- [FixNorm: Dissecting Weight Decay for Training Deep Neural Networks](https://mlsanity.com/paper/2103.15345/) (2021)
- [AutoML in Heavily Constrained Applications](https://mlsanity.com/paper/2306.16913/) (2023)
- [Towards Automated Design of Bayesian Optimization via Exploratory Landscape Analysis](https://mlsanity.com/paper/2211.09678/) (2022)
- [QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks](https://mlsanity.com/paper/2604.17842/) (2026)
- [Preference-Optimal Multi-Metric Weighting for Parallel Coordinate Plots](https://mlsanity.com/paper/2507.02905/) (2025)
- [Gradient-Informed Bayesian and Interior Point Optimization for Efficient Inverse Design in Nanophotonics](https://mlsanity.com/paper/2602.18148/) (2026)
- [mlr3mbo: Bayesian Optimization in R](https://mlsanity.com/paper/2603.29730/) (2026)

## Abstract (arXiv)

Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning. Although it is well-accepted that HPO is crucial to obtain well-performing machine learning models, tuning BO's own hyperparameters is often neglected. In this paper, we empirically study the impact of optimizing BO's own hyperparameters and the transferability of the found settings using a wide range of benchmarks, including artificial functions, HPO and HPO combined with neural architecture search. In particular, we show (i) that tuning can improve the any-time performance of different BO approaches, that optimized BO settings also perform well (ii) on similar problems and (iii) partially even on problems from other problem families, and (iv) which BO hyperparameters are most important.