# An effective algorithm for hyperparameter optimization of neural networks

arXiv: https://arxiv.org/abs/1705.08520 · HTML: https://mlsanity.com/paper/1705.08520/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2017.

## Built on by (most cited first, 5 of 5)

- [Search Algorithms for Automated Hyper-Parameter Tuning](https://mlsanity.com/paper/2104.14677/) (2021)
- [Generalization of CNNs on Relational Reasoning with Bar Charts](https://mlsanity.com/paper/2503.00086/) (2025)
- [Deep P-Spline: Theory, Fast Tuning, and Application](https://mlsanity.com/paper/2501.01376/) (2025)
- [Beyond Trend Following: Deep Learning for Market Trend Prediction](https://mlsanity.com/paper/2407.13685/) (2024)
- [Efficient Deep Learning Board: Training Feedback Is Not All You Need](https://mlsanity.com/paper/2410.14743/) (2024)

## Abstract (arXiv)

A major challenge in designing neural network (NN) systems is to determine the best structure and parameters for the network given the data for the machine learning problem at hand. Examples of parameters are the number of layers and nodes, the learning rates, and the dropout rates. Typically, these parameters are chosen based on heuristic rules and manually fine-tuned, which may be very time-consuming, because evaluating the performance of a single parametrization of the NN may require several hours. This paper addresses the problem of choosing appropriate parameters for the NN by formulating it as a box-constrained mathematical optimization problem, and applying a derivative-free optimization tool that automatically and effectively searches the parameter space. The optimization tool employs a radial basis function model of the objective function (the prediction accuracy of the NN) to accelerate the discovery of configurations yielding high accuracy. Candidate configurations explored by the algorithm are trained to a small number of epochs, and only the most promising candidates receive full training. The performance of the proposed methodology is assessed on benchmark sets and in the context of predicting drug-drug interactions, showing promising results. The optimization tool used in this paper is open-source.