# HyperBO+: Pre-training a universal prior for Bayesian optimization with hierarchical Gaussian processes

arXiv: https://arxiv.org/abs/2212.10538 · HTML: https://mlsanity.com/paper/2212.10538/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Max-value Entropy Search for Efficient Bayesian Optimization](https://mlsanity.com/paper/1703.01968/) (2017)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation](https://mlsanity.com/paper/2012.03826/) (2020)
- [Towards Learning Universal Hyperparameter Optimizers with Transformers](https://mlsanity.com/paper/2205.13320/) (2022)
- [HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML](https://mlsanity.com/paper/2106.06257/) (2021)
- [Few-Shot Bayesian Optimization with Deep Kernel Surrogates](https://mlsanity.com/paper/2101.07667/) (2021)
- [Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization](https://mlsanity.com/paper/1904.02642/) (2019)
- [Pre-trained Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2109.08215/) (2021)
- [Misspecified Gaussian Process Bandit Optimization](https://mlsanity.com/paper/2111.05008/) (2021)
- [Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior](https://mlsanity.com/paper/1811.09558/) (2018)
- [Learning to guide task and motion planning using score-space representation](https://mlsanity.com/paper/1807.09962/) (2018)
- [Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posed](https://mlsanity.com/paper/2203.09179/) (2022)
- [The Human Kernel](https://mlsanity.com/paper/1510.07389/) (2015)
- [Focused Model-Learning and Planning for Non-Gaussian Continuous State-Action Systems](https://mlsanity.com/paper/1607.07762/) (2016)
- [A Bayesian/Information Theoretic Model of Bias Learning](https://mlsanity.com/paper/1911.06129/) (2019)

## Built on by (most cited first, 4 of 4)

- [Looping in the Human Collaborative and Explainable Bayesian Optimization](https://mlsanity.com/paper/2310.17273/) (2023)
- [Transfer Learning for Bayesian Optimization on Heterogeneous Search Spaces](https://mlsanity.com/paper/2309.16597/) (2023)
- [Large Scale Multi-Task Bayesian Optimization with Large Language Models](https://mlsanity.com/paper/2503.08131/) (2025)
- [Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization](https://mlsanity.com/paper/2605.04895/) (2026)

## Abstract (arXiv)

Bayesian optimization (BO), while proved highly effective for many black-box function optimization tasks, requires practitioners to carefully select priors that well model their functions of interest. Rather than specifying by hand, researchers have investigated transfer learning based methods to automatically learn the priors, e.g. multi-task BO (Swersky et al., 2013), few-shot BO (Wistuba and Grabocka, 2021) and HyperBO (Wang et al., 2022). However, those prior learning methods typically assume that the input domains are the same for all tasks, weakening their ability to use observations on functions with different domains or generalize the learned priors to BO on different search spaces. In this work, we present HyperBO+: a pre-training approach for hierarchical Gaussian processes that enables the same prior to work universally for Bayesian optimization on functions with different domains. We propose a two-step pre-training method and analyze its appealing asymptotic properties and benefits to BO both theoretically and empirically. On real-world hyperparameter tuning tasks that involve multiple search spaces, we demonstrate that HyperBO+ is able to generalize to unseen search spaces and achieves lower regrets than competitive baselines.