# Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning

arXiv: https://arxiv.org/abs/1909.12552 · HTML: https://mlsanity.com/paper/1909.12552/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2019.

## Built on by (most cited first, 30 of 50)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML](https://mlsanity.com/paper/2106.06257/) (2021)
- [πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization](https://mlsanity.com/paper/2204.11051/) (2022)
- [End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes](https://mlsanity.com/paper/2305.15930/) (2023)
- [PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning](https://mlsanity.com/paper/2306.12370/) (2023)
- [Artificial intelligence to advance Earth observation: : A review of models, recent trends, and pathways forward](https://mlsanity.com/paper/2305.08413/) (2023)
- [Transfer Learning for Bayesian Optimization: A Survey](https://mlsanity.com/paper/2302.05927/) (2023)
- [Sequential Large Language Model-Based Hyper-parameter Optimization](https://mlsanity.com/paper/2410.20302/) (2024)
- [FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch](https://mlsanity.com/paper/2406.04824/) (2024)
- [A General Framework for User-Guided Bayesian Optimization](https://mlsanity.com/paper/2311.14645/) (2023)
- [PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces](https://mlsanity.com/paper/2304.10255/) (2023)
- [FLoRA: Single-shot Hyper-parameter Optimization for Federated Learning](https://mlsanity.com/paper/2112.08524/) (2021)
- [Reinforced In-Context Black-Box Optimization](https://mlsanity.com/paper/2402.17423/) (2024)
- [Transfer Learning based Search Space Design for Hyperparameter Tuning](https://mlsanity.com/paper/2206.02511/) (2022)
- [A multi-objective perspective on jointly tuning hardware and hyperparameters](https://mlsanity.com/paper/2106.05680/) (2021)
- [Bayesian Active Learning by Disagreements: A Geometric Perspective](https://mlsanity.com/paper/2105.02543/) (2021)
- [JUMBO: Scalable Multi-task Bayesian Optimization using Offline Data](https://mlsanity.com/paper/2106.00942/) (2021)
- [On Provably Robust Meta-Bayesian Optimization](https://mlsanity.com/paper/2206.06872/) (2022)
- [Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/2110.10461/) (2021)
- [TransBO: Hyperparameter Optimization via Two-Phase Transfer Learning](https://mlsanity.com/paper/2206.02663/) (2022)
- [Speeding Up Multi-Objective Hyperparameter Optimization by Task Similarity-Based Meta-Learning for the Tree-Structured Parzen Estimator](https://mlsanity.com/paper/2212.06751/) (2022)
- [Monte Carlo Tree Search based Space Transfer for Black-box Optimization](https://mlsanity.com/paper/2412.07186/) (2024)
- [Show Your Work with Confidence: Confidence Bands for Tuning Curves](https://mlsanity.com/paper/2311.09480/) (2023)
- [A resource-efficient method for repeated HPO and NAS problems](https://mlsanity.com/paper/2103.16111/) (2021)
- [ODBO: Bayesian Optimization with Search Space Prescreening for Directed Protein Evolution](https://mlsanity.com/paper/2205.09548/) (2022)
- [Amortized Auto-Tuning: Cost-Efficient Bayesian Transfer Optimization for Hyperparameter Recommendation](https://mlsanity.com/paper/2106.09179/) (2021)
- [HPC Storage Service Autotuning Using Variational-Autoencoder-Guided Asynchronous Bayesian Optimization](https://mlsanity.com/paper/2210.00798/) (2022)
- [On the Hyperparameter Loss Landscapes of Machine Learning Models: An Exploratory Study](https://mlsanity.com/paper/2311.14014/) (2023)
- [Efficient Task Transfer for HLS DSE](https://mlsanity.com/paper/2408.13270/) (2024)
- [Hyperparameter Optimization with Differentiable Metafeatures](https://mlsanity.com/paper/2102.03776/) (2021)

## Abstract (arXiv)

Bayesian optimization (BO) is a successful methodology to optimize black-box functions that are expensive to evaluate. While traditional methods optimize each black-box function in isolation, there has been recent interest in speeding up BO by transferring knowledge across multiple related black-box functions. In this work, we introduce a method to automatically design the BO search space by relying on evaluations of previous black-box functions. We depart from the common practice of defining a set of arbitrary search ranges a priori by considering search space geometries that are learned from historical data. This simple, yet effective strategy can be used to endow many existing BO methods with transfer learning properties. Despite its simplicity, we show that our approach considerably boosts BO by reducing the size of the search space, thus accelerating the optimization of a variety of black-box optimization problems. In particular, the proposed approach combined with random search results in a parameter-free, easy-to-implement, robust hyperparameter optimization strategy. We hope it will constitute a natural baseline for further research attempting to warm-start BO.