# A Comparative Study of Hyperparameter Tuning Methods

arXiv: https://arxiv.org/abs/2408.16425 · HTML: https://mlsanity.com/paper/2408.16425/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 41 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 7 of 7)

- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Few-Shot Bayesian Optimization with Deep Kernel Surrogates](https://mlsanity.com/paper/2101.07667/) (2021)
- [Incorporating Expert Prior in Bayesian Optimisation via Space Warping](https://mlsanity.com/paper/2003.12250/) (2020)
- [Risk-averse Heteroscedastic Bayesian Optimization](https://mlsanity.com/paper/2111.03637/) (2021)
- [A Robust Asymmetric Kernel Function for Bayesian Optimization, with Application to Image Defect Detection in Manufacturing Systems](https://mlsanity.com/paper/2109.10898/) (2021)
- [Self-Optimizing Grinding Machines using Gaussian Process Models and Constrained Bayesian Optimization](https://mlsanity.com/paper/2006.05360/) (2020)
- [Genetic Algorithm for Multi-Objective Optimization of Container Allocation in Cloud Architecture](https://mlsanity.com/paper/2401.12698/) (2024)

## Abstract (arXiv)

The study emphasizes the challenge of finding the optimal trade-off between bias and variance, especially as hyperparameter optimization increases in complexity. Through empirical analysis, three hyperparameter tuning algorithms Tree-structured Parzen Estimator (TPE), Genetic Search, and Random Search are evaluated across regression and classification tasks. The results show that nonlinear models, with properly tuned hyperparameters, significantly outperform linear models. Interestingly, Random Search excelled in regression tasks, while TPE was more effective for classification tasks. This suggests that there is no one-size-fits-all solution, as different algorithms perform better depending on the task and model type. The findings underscore the importance of selecting the appropriate tuning method and highlight the computational challenges involved in optimizing machine learning models, particularly as search spaces expand.