mlsanityData to 7 Oct 2026

Experimental Investigation and Evaluation of Model-based Hyperparameter Optimization

arXiv 2107.08761 ↗First posted 19 Jul 2021cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Machine learning algorithms such as random forests or xgboost are gaining more importance and are increasingly incorporated into production processes in order to enable comprehensive digitization and, if possible, automation of processes. Hyperparameters of these algorithms used have to be set appropriately, which can be referred to as hyperparameter tuning or optimization. Based on the concept of tunability, this article presents an overview of theoretical and practical results for popular machine learning algorithms. This overview is accompanied by an experimental analysis of 30 hyperparameters from six relevant machine learning algorithms. In particular, it provides (i) a survey of important hyperparameters, (ii) two parameter tuning studies, and (iii) one extensive global parameter tuning study, as well as (iv) a new way, based on consensus ranking, to analyze results from multiple algorithms. The R package mlr is used as a uniform interface to the machine learning models. The R package SPOT is used to perform the actual tuning (optimization). All additional code is provided together with this paper.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 64 references that are in the map. First posted on arXiv 19 Jul 2021.

Builds on

20 of 64 references.

References · 20

  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2014OpenML: networked science in machine learningcited by 523
  3. 2020On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practicecited by 181
  4. 2015ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and Rcited by 171
  5. 2018Tunability: Importance of Hyperparameters of Machine Learning Algorithmscited by 98
  6. 2014ParamILS: An Automatic Algorithm Configuration Frameworkcited by 88
  7. 2018Hyperparameters and Tuning Strategies for Random Forestcited by 83
  8. 2018How Complex is your classification problem? A survey on measuring classification complexitycited by 66
  9. 2017Hyperparameter Importance Across Datasetscited by 62
  10. 2014Understanding Random Forests: From Theory to Practicecited by 57
  11. 2019Greed is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisationcited by 47
  12. 2017To tune or not to tune the number of trees in random forest?cited by 33
8 more
  1. 2017Distance and Similarity Measures Effect on the Performance of K-Nearest Neighbor Classifier -- A Reviewcited by 25
  2. 2018Gradient and Newton Boosting for Classification and Regressioncited by 14
  3. 2018Performance evaluation and hyperparameter tuning of statistical and machine-learning models using spatial datacited by 11
  4. 2019An Experimental Comparison of Old and New Decision Tree Algorithmscited by 7
  5. 2018Automatic Gradient Boostingcited by 6
  6. 2020Expected Improvement versus Predicted Value in Surrogate-Based Optimizationcited by 6
  7. 2017The True Destination of EGO is Multi-local Optimizationcited by 5
  8. 2019A XGBoost risk model via feature selection and Bayesian hyper-parameter optimizationcited by 5

Built on by

Not cited yet.