# Hyperboost: Hyperparameter Optimization by Gradient Boosting surrogate models

arXiv: https://arxiv.org/abs/2101.02289 · HTML: https://mlsanity.com/paper/2101.02289/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 12 of 12)

- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [CatBoost: gradient boosting with categorical features support](https://mlsanity.com/paper/1810.11363/) (2018)
- [Bayesian Optimization in a Billion Dimensions via Random Embeddings](https://mlsanity.com/paper/1301.1942/) (2013)
- [High Dimensional Bayesian Optimisation and Bandits via Additive Models](https://mlsanity.com/paper/1503.01673/) (2015)
- [Batched Large-scale Bayesian Optimization in High-dimensional Spaces](https://mlsanity.com/paper/1706.01445/) (2017)
- [On the State of the Art of Evaluation in Neural Language Models](https://mlsanity.com/paper/1707.05589/) (2017)
- [CMA-ES for Hyperparameter Optimization of Deep Neural Networks](https://mlsanity.com/paper/1604.07269/) (2016)
- [Data-driven Advice for Applying Machine Learning to Bioinformatics Problems](https://mlsanity.com/paper/1708.05070/) (2017)
- [Single-Model Uncertainties for Deep Learning](https://mlsanity.com/paper/1811.00908/) (2018)
- [FLASH: Fast Bayesian Optimization for Data Analytic Pipelines](https://mlsanity.com/paper/1602.06468/) (2016)

## Built on by (most cited first, 4 of 4)

- [LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithms](https://mlsanity.com/paper/2505.21034/) (2025)
- [Benchmarking state-of-the-art gradient boosting algorithms for classification](https://mlsanity.com/paper/2305.17094/) (2023)
- [Automating Data Science Pipelines with Tensor Completion](https://mlsanity.com/paper/2410.06408/) (2024)
- [Enhancing Performance and Calibration in Quantile Hyperparameter Optimization](https://mlsanity.com/paper/2509.17051/) (2025)

## Abstract (arXiv)

Bayesian Optimization is a popular tool for tuning algorithms in automatic machine learning (AutoML) systems. Current state-of-the-art methods leverage Random Forests or Gaussian processes to build a surrogate model that predicts algorithm performance given a certain set of hyperparameter settings. In this paper, we propose a new surrogate model based on gradient boosting, where we use quantile regression to provide optimistic estimates of the performance of an unobserved hyperparameter setting, and combine this with a distance metric between unobserved and observed hyperparameter settings to help regulate exploration. We demonstrate empirically that the new method is able to outperform some state-of-the art techniques across a reasonable sized set of classification problems.