# Using Sequential Statistical Tests for Efficient Hyperparameter Tuning

arXiv: https://arxiv.org/abs/2112.12438 · HTML: https://mlsanity.com/paper/2112.12438/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 44 references that are in the map.
- First posted on arXiv 23 Dec 2021.

## Builds on (most cited first, 8 of 8)

- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Hyper-Parameter Optimization: A Review of Algorithms and Applications](https://mlsanity.com/paper/2003.05689/) (2020)
- [CMA-ES for Hyperparameter Optimization of Deep Neural Networks](https://mlsanity.com/paper/1604.07269/) (2016)
- [Automated Machine Learning with Monte-Carlo Tree Search](https://mlsanity.com/paper/1906.00170/) (2019)
- [Travel Time Prediction using Tree-Based Ensembles](https://mlsanity.com/paper/2005.13818/) (2020)
- [An Uncertainty-Aware, Shareable and Transparent Neural Network Architecture for Brain-Age Modeling](https://mlsanity.com/paper/2107.07977/) (2021)

## Built on by (most cited first, 3 of 3)

- [Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization](https://mlsanity.com/paper/2405.15393/) (2024)
- [Overtuning in Hyperparameter Optimization](https://mlsanity.com/paper/2506.19540/) (2025)
- [Adaptive Resampling with Bootstrap for Noisy Multi-Objective Optimization Problems](https://mlsanity.com/paper/2503.21495/) (2025)

## Abstract (arXiv)

Hyperparameter tuning is one of the the most time-consuming parts in machine learning. Despite the existence of modern optimization algorithms that minimize the number of evaluations needed, evaluations of a single setting may still be expensive. Usually a resampling technique is used, where the machine learning method has to be fitted a fixed number of k times on different training datasets. The respective mean performance of the k fits is then used as performance estimator. Many hyperparameter settings could be discarded after less than k resampling iterations if they are clearly inferior to high-performing settings. However, resampling is often performed until the very end, wasting a lot of computational effort. To this end, we propose the Sequential Random Search (SQRS) which extends the regular random search algorithm by a sequential testing procedure aimed at detecting and eliminating inferior parameter configurations early. We compared our SQRS with regular random search using multiple publicly available regression and classification datasets. Our simulation study showed that the SQRS is able to find similarly well-performing parameter settings while requiring noticeably fewer evaluations. Our results underscore the potential for integrating sequential tests into hyperparameter tuning.