# Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters

arXiv: https://arxiv.org/abs/2201.05978 · HTML: https://mlsanity.com/paper/2201.05978/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 69 references that are in the map.
- First posted on arXiv 16 Jan 2022.

## Builds on (most cited first, 13 of 13)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Hyperparameter optimization with approximate gradient](https://mlsanity.com/paper/1602.02355/) (2016)
- [On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice](https://mlsanity.com/paper/2007.15745/) (2020)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Hyper-Parameter Optimization: A Review of Algorithms and Applications](https://mlsanity.com/paper/2003.05689/) (2020)
- [Hyp-RL : Hyperparameter Optimization by Reinforcement Learning](https://mlsanity.com/paper/1906.11527/) (2019)
- [MANGO: A Python Library for Parallel Hyperparameter Tuning](https://mlsanity.com/paper/2005.11394/) (2020)
- [Blockchain Networks: Data Structures of Bitcoin, Monero, Zcash, Ethereum, Ripple and Iota](https://mlsanity.com/paper/2103.08712/) (2021)
- [Efficient Ranking and Selection in Parallel Computing Environments](https://mlsanity.com/paper/1506.04986/) (2015)
- [A Note on the Stochastic Ruler Method for Discrete Simulation Optimization](https://mlsanity.com/paper/2010.06909/) (2020)

## Abstract (arXiv)

Machine learning (ML) methods are used in most technical areas such as image recognition, product recommendation, financial analysis, medical diagnosis, and predictive maintenance. An important aspect of implementing ML methods involves controlling the learning process for the ML method so as to maximize the performance of the method under consideration. Hyperparameter tuning is the process of selecting a suitable set of ML method parameters that control its learning process. In this work, we demonstrate the use of discrete simulation optimization methods such as ranking and selection (R&S) and random search for identifying a hyperparameter set that maximizes the performance of a ML method. Specifically, we use the KN R&S method and the stochastic ruler random search method and one of its variations for this purpose. We also construct the theoretical basis for applying the KN method, which determines the optimal solution with a statistical guarantee via solution space enumeration. In comparison, the stochastic ruler method asymptotically converges to global optima and incurs smaller computational overheads. We demonstrate the application of these methods to a wide variety of machine learning models, including deep neural network models used for time series prediction and image classification. We benchmark our application of these methods with state-of-the-art hyperparameter optimization libraries such as hyperopt and mango. The KN method consistently outperforms hyperopt's random search (RS) and Tree of Parzen Estimators (TPE) methods. The stochastic ruler method outperforms the hyperopt RS method and offers statistically comparable performance with respect to hyperopt's TPE method and the mango algorithm.