# Experimental Investigation and Evaluation of Model-based Hyperparameter Optimization

arXiv: https://arxiv.org/abs/2107.08761 · HTML: https://mlsanity.com/paper/2107.08761/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 64 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 20 of 20)

- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice](https://mlsanity.com/paper/2007.15745/) (2020)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [ParamILS: An Automatic Algorithm Configuration Framework](https://mlsanity.com/paper/1401.3492/) (2014)
- [Hyperparameters and Tuning Strategies for Random Forest](https://mlsanity.com/paper/1804.03515/) (2018)
- [How Complex is your classification problem? A survey on measuring classification complexity](https://mlsanity.com/paper/1808.03591/) (2018)
- [Hyperparameter Importance Across Datasets](https://mlsanity.com/paper/1710.04725/) (2017)
- [Understanding Random Forests: From Theory to Practice](https://mlsanity.com/paper/1407.7502/) (2014)
- [Greed is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisation](https://mlsanity.com/paper/1911.12809/) (2019)
- [To tune or not to tune the number of trees in random forest?](https://mlsanity.com/paper/1705.05654/) (2017)
- [Distance and Similarity Measures Effect on the Performance of K-Nearest Neighbor Classifier -- A Review](https://mlsanity.com/paper/1708.04321/) (2017)
- [Gradient and Newton Boosting for Classification and Regression](https://mlsanity.com/paper/1808.03064/) (2018)
- [Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial data](https://mlsanity.com/paper/1803.11266/) (2018)
- [An Experimental Comparison of Old and New Decision Tree Algorithms](https://mlsanity.com/paper/1911.03054/) (2019)
- [Automatic Gradient Boosting](https://mlsanity.com/paper/1807.03873/) (2018)
- [Expected Improvement versus Predicted Value in Surrogate-Based Optimization](https://mlsanity.com/paper/2001.02957/) (2020)
- [The True Destination of EGO is Multi-local Optimization](https://mlsanity.com/paper/1704.05724/) (2017)
- [A XGBoost risk model via feature selection and Bayesian hyper-parameter optimization](https://mlsanity.com/paper/1901.08433/) (2019)

## Abstract (arXiv)

Machine learning algorithms such as random forests or xgboost are gaining more importance and are increasingly incorporated into production processes in order to enable comprehensive digitization and, if possible, automation of processes. Hyperparameters of these algorithms used have to be set appropriately, which can be referred to as hyperparameter tuning or optimization. Based on the concept of tunability, this article presents an overview of theoretical and practical results for popular machine learning algorithms. This overview is accompanied by an experimental analysis of 30 hyperparameters from six relevant machine learning algorithms. In particular, it provides (i) a survey of important hyperparameters, (ii) two parameter tuning studies, and (iii) one extensive global parameter tuning study, as well as (iv) a new way, based on consensus ranking, to analyze results from multiple algorithms. The R package mlr is used as a uniform interface to the machine learning models. The R package SPOT is used to perform the actual tuning (optimization). All additional code is provided together with this paper.