# Fast Hyperparameter Tuning using Bayesian Optimization with Directional Derivatives

arXiv: https://arxiv.org/abs/1902.02416 · HTML: https://mlsanity.com/paper/1902.02416/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2019.

## Built on by (most cited first, 3 of 3)

- [Sequential Large Language Model-Based Hyper-parameter Optimization](https://mlsanity.com/paper/2410.20302/) (2024)
- [Fast Bayesian Optimization of Needle-in-a-Haystack Problems using Zooming Memory-Based Initialization (ZoMBI)](https://mlsanity.com/paper/2208.13771/) (2022)
- [Interval Prediction of Annual Average Daily Traffic on Local Roads via Quantile Random Forest with High-Dimensional Spatial Data](https://mlsanity.com/paper/2510.18548/) (2025)

## Abstract (arXiv)

In this paper we develop a Bayesian optimization based hyperparameter tuning framework inspired by statistical learning theory for classifiers. We utilize two key facts from PAC learning theory; the generalization bound will be higher for a small subset of data compared to the whole, and the highest accuracy for a small subset of data can be achieved with a simple model. We initially tune the hyperparameters on a small subset of training data using Bayesian optimization. While tuning the hyperparameters on the whole training data, we leverage the insights from the learning theory to seek more complex models. We realize this by using directional derivative signs strategically placed in the hyperparameter search space to seek a more complex model than the one obtained with small data. We demonstrate the performance of our method on the tasks of tuning the hyperparameters of several machine learning algorithms.