# Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates

arXiv: https://arxiv.org/abs/1607.08316 · HTML: https://mlsanity.com/paper/1607.08316/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2016.

## Built on by (most cited first, 7 of 7)

- [Adaptive Gradient Methods with Local Guarantees](https://mlsanity.com/paper/2203.01400/) (2022)
- [Fully Elman Neural Network: A Novel Deep Recurrent Neural Network Optimized by an Improved Harris Hawks Algorithm for Classification of Pulmonary Arterial Wedge Pressure](https://mlsanity.com/paper/2301.07710/) (2023)
- [HomOpt: A Homotopy-Based Hyperparameter Optimization Method](https://mlsanity.com/paper/2308.03317/) (2023)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [Landscape-Sketch-Step: An AI/ML-Based Metaheuristic for Surrogate Optimization Problems](https://mlsanity.com/paper/2309.07936/) (2023)
- [A Deep Neural Network -- Mechanistic Hybrid Model to Predict Pharmacokinetics in Rat](https://mlsanity.com/paper/2310.09167/) (2023)
- [Time Series Imputation with Multivariate Radial Basis Function Neural Network](https://mlsanity.com/paper/2407.17040/) (2024)

## Abstract (arXiv)

Automatically searching for optimal hyperparameter configurations is of crucial importance for applying deep learning algorithms in practice. Recently, Bayesian optimization has been proposed for optimizing hyperparameters of various machine learning algorithms. Those methods adopt probabilistic surrogate models like Gaussian processes to approximate and minimize the validation error function of hyperparameter values. However, probabilistic surrogates require accurate estimates of sufficient statistics (e.g., covariance) of the error distribution and thus need many function evaluations with a sizeable number of hyperparameters. This makes them inefficient for optimizing hyperparameters of deep learning algorithms, which are highly expensive to evaluate. In this work, we propose a new deterministic and efficient hyperparameter optimization method that employs radial basis functions as error surrogates. The proposed mixed integer algorithm, called HORD, searches the surrogate for the most promising hyperparameter values through dynamic coordinate search and requires many fewer function evaluations. HORD does well in low dimensions but it is exceptionally better in higher dimensions. Extensive evaluations on MNIST and CIFAR-10 for four deep neural networks demonstrate HORD significantly outperforms the well-established Bayesian optimization methods such as GP, SMAC, and TPE. For instance, on average, HORD is more than 6 times faster than GP-EI in obtaining the best configuration of 19 hyperparameters.