# Transfer Learning based Search Space Design for Hyperparameter Tuning

arXiv: https://arxiv.org/abs/2206.02511 · HTML: https://mlsanity.com/paper/2206.02511/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 6 Jun 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning](https://mlsanity.com/paper/1909.12552/) (2019)
- [OpenBox: A Generalized Black-box Optimization Service](https://mlsanity.com/paper/2106.00421/) (2021)
- [OBOE: Collaborative Filtering for AutoML Model Selection](https://mlsanity.com/paper/1808.03233/) (2018)
- [Efficient Automatic CASH via Rising Bandits](https://mlsanity.com/paper/2012.04371/) (2020)
- [VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition](https://mlsanity.com/paper/2107.08861/) (2021)
- [Autotune: A Derivative-free Optimization Framework for Hyperparameter Tuning](https://mlsanity.com/paper/1804.07824/) (2018)
- [MFES-HB: Efficient Hyperband with Multi-Fidelity Quality Measurements](https://mlsanity.com/paper/2012.03011/) (2020)
- [Warmstarting of Model-based Algorithm Configuration](https://mlsanity.com/paper/1709.04636/) (2017)
- [Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale](https://mlsanity.com/paper/2201.06834/) (2022)
- [Amazon SageMaker Automatic Model Tuning: Scalable Gradient-Free Optimization](https://mlsanity.com/paper/2012.08489/) (2020)
- [Learning to Warm-Start Bayesian Hyperparameter Optimization](https://mlsanity.com/paper/1710.06219/) (2017)

## Built on by (most cited first, 13 of 13)

- [Transfer Learning for Bayesian Optimization: A Survey](https://mlsanity.com/paper/2302.05927/) (2023)
- [Sequential Large Language Model-Based Hyper-parameter Optimization](https://mlsanity.com/paper/2410.20302/) (2024)
- [Reinforced In-Context Black-Box Optimization](https://mlsanity.com/paper/2402.17423/) (2024)
- [Monte Carlo Tree Search based Space Transfer for Black-box Optimization](https://mlsanity.com/paper/2412.07186/) (2024)
- [A Unified and Efficient Coordinating Framework for Autonomous DBMS Tuning](https://mlsanity.com/paper/2303.05710/) (2023)
- [On the Hyperparameter Loss Landscapes of Machine Learning Models: An Exploratory Study](https://mlsanity.com/paper/2311.14014/) (2023)
- [Principled Architecture-aware Scaling of Hyperparameters](https://mlsanity.com/paper/2402.17440/) (2024)
- [Rover: An online Spark SQL tuning service via generalized transfer learning](https://mlsanity.com/paper/2302.04046/) (2023)
- [None To Optima in Few Shots: Bayesian Optimization with MDP Priors](https://mlsanity.com/paper/2511.01006/) (2025)
- [MALIBO: Meta-learning for Likelihood-free Bayesian Optimization](https://mlsanity.com/paper/2307.03565/) (2023)
- [Towards General and Efficient Online Tuning for Spark](https://mlsanity.com/paper/2309.01901/) (2023)
- [Provable Accelerated Bayesian Optimization with Knowledge Transfer](https://mlsanity.com/paper/2511.03125/) (2025)
- [Efficient End-to-End AutoML via Scalable Search Space Decomposition](https://mlsanity.com/paper/2206.09423/) (2022)

## Abstract (arXiv)

The tuning of hyperparameters becomes increasingly important as machine learning (ML) models have been extensively applied in data mining applications. Among various approaches, Bayesian optimization (BO) is a successful methodology to tune hyper-parameters automatically. While traditional methods optimize each tuning task in isolation, there has been recent interest in speeding up BO by transferring knowledge across previous tasks. In this work, we introduce an automatic method to design the BO search space with the aid of tuning history from past tasks. This simple yet effective approach can be used to endow many existing BO methods with transfer learning capabilities. In addition, it enjoys the three advantages: universality, generality, and safeness. The extensive experiments show that our approach considerably boosts BO by designing a promising and compact search space instead of using the entire space, and outperforms the state-of-the-arts on a wide range of benchmarks, including machine learning and deep learning tuning tasks, and neural architecture search.