# Initial Design Strategies and their Effects on Sequential Model-Based Optimization

arXiv: https://arxiv.org/abs/2003.13826 · HTML: https://mlsanity.com/paper/2003.13826/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2020.

## Built on by (most cited first, 10 of 10)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [A Collection of Deep Learning-based Feature-Free Approaches for Characterizing Single-Objective Continuous Fitness Landscapes](https://mlsanity.com/paper/2204.05752/) (2022)
- [PI is back! Switching Acquisition Functions in Bayesian Optimization](https://mlsanity.com/paper/2211.01455/) (2022)
- [Self-Adjusting Weighted Expected Improvement for Bayesian Optimization](https://mlsanity.com/paper/2306.04262/) (2023)
- [Faster variational quantum algorithms with quantum kernel-based surrogate models](https://mlsanity.com/paper/2211.01134/) (2022)
- [Towards Automated Design of Bayesian Optimization via Exploratory Landscape Analysis](https://mlsanity.com/paper/2211.09678/) (2022)
- [Informed Initialization for Bayesian Optimization and Active Learning](https://mlsanity.com/paper/2510.23681/) (2025)
- [HO-FMN: Hyperparameter Optimization for Fast Minimum-Norm Attacks](https://mlsanity.com/paper/2407.08806/) (2024)
- [mlr3mbo: Bayesian Optimization in R](https://mlsanity.com/paper/2603.29730/) (2026)

## Abstract (arXiv)

Sequential model-based optimization (SMBO) approaches are algorithms for solving problems that require computationally or otherwise expensive function evaluations. The key design principle of SMBO is a substitution of the true objective function by a surrogate, which is used to propose the point(s) to be evaluated next. SMBO algorithms are intrinsically modular, leaving the user with many important design choices. Significant research efforts go into understanding which settings perform best for which type of problems. Most works, however, focus on the choice of the model, the acquisition function, and the strategy used to optimize the latter. The choice of the initial sampling strategy, however, receives much less attention. Not surprisingly, quite diverging recommendations can be found in the literature. We analyze in this work how the size and the distribution of the initial sample influences the overall quality of the efficient global optimization~(EGO) algorithm, a well-known SMBO approach. While, overall, small initial budgets using Halton sampling seem preferable, we also observe that the performance landscape is rather unstructured. We furthermore identify several situations in which EGO performs unfavorably against random sampling. Both observations indicate that an adaptive SMBO design could be beneficial, making SMBO an interesting test-bed for automated algorithm design.