# Behavior of Hyper-Parameters for Selected Machine Learning Algorithms: An Empirical Investigation

arXiv: https://arxiv.org/abs/2211.08536 · HTML: https://mlsanity.com/paper/2211.08536/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 17 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 5 of 5)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning](https://mlsanity.com/paper/2102.13651/) (2021)
- [Importance of Tuning Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/2007.07588/) (2020)
- [Supervised Machine Learning Techniques: An Overview with Applications to Banking](https://mlsanity.com/paper/2008.04059/) (2020)
- [Hyperparameter Importance for Machine Learning Algorithms](https://mlsanity.com/paper/2201.05132/) (2022)

## Abstract (arXiv)

Hyper-parameters (HPs) are an important part of machine learning (ML) model development and can greatly influence performance. This paper studies their behavior for three algorithms: Extreme Gradient Boosting (XGB), Random Forest (RF), and Feedforward Neural Network (FFNN) with structured data. Our empirical investigation examines the qualitative behavior of model performance as the HPs vary, quantifies the importance of each HP for different ML algorithms, and stability of the performance near the optimal region. Based on the findings, we propose a set of guidelines for efficient HP tuning by reducing the search space.