# Transfer Learning for Bayesian Optimization on Heterogeneous Search Spaces

arXiv: https://arxiv.org/abs/2309.16597 · HTML: https://mlsanity.com/paper/2309.16597/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 13 of 13)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Max-value Entropy Search for Efficient Bayesian Optimization](https://mlsanity.com/paper/1703.01968/) (2017)
- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [Towards Learning Universal Hyperparameter Optimizers with Transformers](https://mlsanity.com/paper/2205.13320/) (2022)
- [Bayesian optimization for materials design](https://mlsanity.com/paper/1506.01349/) (2015)
- [HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML](https://mlsanity.com/paper/2106.06257/) (2021)
- [Few-Shot Bayesian Optimization with Deep Kernel Surrogates](https://mlsanity.com/paper/2101.07667/) (2021)
- [Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimization](https://mlsanity.com/paper/1904.02642/) (2019)
- [Pre-trained Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2109.08215/) (2021)
- [Learning compositional models of robot skills for task and motion planning](https://mlsanity.com/paper/2006.06444/) (2020)
- [Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior](https://mlsanity.com/paper/1811.09558/) (2018)
- [Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posed](https://mlsanity.com/paper/2203.09179/) (2022)
- [HyperBO+: Pre-training a universal prior for Bayesian optimization with hierarchical Gaussian processes](https://mlsanity.com/paper/2212.10538/) (2022)

## Built on by (most cited first, 9 of 9)

- [QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?](https://mlsanity.com/paper/2503.22674/) (2025)
- [OmniPred: Language Models as Universal Regressors](https://mlsanity.com/paper/2402.14547/) (2024)
- [Language Model Embeddings Can Be Sufficient for Bayesian Optimization](https://mlsanity.com/paper/2410.10190/) (2024)
- [Monte Carlo Tree Search based Space Transfer for Black-box Optimization](https://mlsanity.com/paper/2412.07186/) (2024)
- [Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings](https://mlsanity.com/paper/2506.07109/) (2025)
- [Amortized Multi-Objective Optimization Across Tasks with Generative Solution Modeling](https://mlsanity.com/paper/2511.09598/) (2025)
- [Sample-Efficient Bayesian Optimization with Transfer Learning for Heterogeneous Search Spaces](https://mlsanity.com/paper/2409.05325/) (2024)
- [Transfer Learning in Bayesian Optimization for Aircraft Design](https://mlsanity.com/paper/2603.28999/) (2026)
- [Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization](https://mlsanity.com/paper/2605.04895/) (2026)

## Abstract (arXiv)

Bayesian optimization (BO) is a popular black-box function optimization method, which makes sequential decisions based on a Bayesian model, typically a Gaussian process (GP), of the function. To ensure the quality of the model, transfer learning approaches have been developed to automatically design GP priors by learning from observations on "training" functions. These training functions are typically required to have the same domain as the "test" function (black-box function to be optimized). In this paper, we introduce MPHD, a model pre-training method on heterogeneous domains, which uses a neural net mapping from domain-specific contexts to specifications of hierarchical GPs. MPHD can be seamlessly integrated with BO to transfer knowledge across heterogeneous search spaces. Our theoretical and empirical results demonstrate the validity of MPHD and its superior performance on challenging black-box function optimization tasks.