# High Per Parameter: A Large-Scale Study of Hyperparameter Tuning for Machine Learning Algorithms

arXiv: https://arxiv.org/abs/2207.06028 · HTML: https://mlsanity.com/paper/2207.06028/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 20 references that are in the map.
- First posted on arXiv 13 Jul 2022.

## Builds on (most cited first, 8 of 8)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [PMLB v1.0: An open source dataset collection for benchmarking machine learning methods](https://mlsanity.com/paper/2012.00058/) (2020)
- [Importance of Tuning Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/2007.07588/) (2020)
- [Neural Networks with A La Carte Selection of Activation Functions](https://mlsanity.com/paper/2206.12166/) (2022)

## Built on by (most cited first, 1 of 1)

- [Target Variable Engineering](https://mlsanity.com/paper/2310.09440/) (2023)

## Abstract (arXiv)

Hyperparameters in machine learning (ML) have received a fair amount of attention, and hyperparameter tuning has come to be regarded as an important step in the ML pipeline. But just how useful is said tuning? While smaller-scale experiments have been previously conducted, herein we carry out a large-scale investigation, specifically, one involving 26 ML algorithms, 250 datasets (regression and both binary and multinomial classification), 6 score metrics, and 28,857,600 algorithm runs. Analyzing the results we conclude that for many ML algorithms we should not expect considerable gains from hyperparameter tuning on average, however, there may be some datasets for which default hyperparameters perform poorly, this latter being truer for some algorithms than others. By defining a single hp_score value, which combines an algorithm's accumulated statistics, we are able to rank the 26 ML algorithms from those expected to gain the most from hyperparameter tuning to those expected to gain the least. We believe such a study may serve ML practitioners at large.