# Bayesian Optimization for Simultaneous Selection of Machine Learning Algorithms and Hyperparameters on Shared Latent Space

arXiv: https://arxiv.org/abs/2502.09329 · HTML: https://mlsanity.com/paper/2502.09329/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science](https://mlsanity.com/paper/1603.06212/) (2016)
- [Pre-trained Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2109.08215/) (2021)
- [Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders](https://mlsanity.com/paper/1804.10469/) (2018)
- [An ADMM Based Framework for AutoML Pipeline Configuration](https://mlsanity.com/paper/1905.00424/) (2019)
- [Efficient Automatic CASH via Rising Bandits](https://mlsanity.com/paper/2012.04371/) (2020)
- [Bayesian Optimization for Categorical and Category-Specific Continuous Inputs](https://mlsanity.com/paper/1911.12473/) (2019)
- [VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition](https://mlsanity.com/paper/2107.08861/) (2021)
- [Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces](https://mlsanity.com/paper/1409.4011/) (2014)
- [Latent Space Bayesian Optimization with Latent Data Augmentation for Enhanced Exploration](https://mlsanity.com/paper/2302.02399/) (2023)
- [RankML: a Meta Learning-Based Approach for Pre-Ranking Machine Learning Pipelines](https://mlsanity.com/paper/1911.00108/) (2019)
- [Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Multiple Heterogeneous Datasets](https://mlsanity.com/paper/2308.04660/) (2023)

## Abstract (arXiv)

Selecting the optimal combination of a machine learning (ML) algorithm and its hyper-parameters is crucial for the development of high-performance ML systems. However, since the combination of ML algorithms and hyper-parameters is enormous, the exhaustive validation requires a significant amount of time. Many existing studies use Bayesian optimization (BO) for accelerating the search. On the other hand, a significant difficulty is that, in general, there exists a different hyper-parameter space for each one of candidate ML algorithms. BO-based approaches typically build a surrogate model independently for each hyper-parameter space, by which sufficient observations are required for all candidate ML algorithms. In this study, our proposed method embeds different hyper-parameter spaces into a shared latent space, in which a surrogate multi-task model for BO is estimated. This approach can share information of observations from different ML algorithms by which efficient optimization is expected with a smaller number of total observations. We further propose the pre-training of the latent space embedding with an adversarial regularization, and a ranking model for selecting an effective pre-trained embedding for a given target dataset. Our empirical study demonstrates effectiveness of the proposed method through datasets from OpenML.