# Bayesian Optimization over Hybrid Spaces

arXiv: https://arxiv.org/abs/2106.04682 · HTML: https://mlsanity.com/paper/2106.04682/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 56 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Batched Large-scale Bayesian Optimization in High-dimensional Spaces](https://mlsanity.com/paper/1706.01445/) (2017)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [Max-value Entropy Search for Multi-Objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/2009.01721/) (2020)
- [Combinatorial Bayesian Optimization using the Graph Cartesian Product](https://mlsanity.com/paper/1902.00448/) (2019)
- [An Easy-to-use Real-world Multi-objective Optimization Problem Suite](https://mlsanity.com/paper/2009.12867/) (2020)
- [Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes](https://mlsanity.com/paper/1706.03673/) (2017)
- [Bayesian Optimization of Combinatorial Structures](https://mlsanity.com/paper/1806.08838/) (2018)
- [Bayesian Optimisation over Multiple Continuous and Categorical Inputs](https://mlsanity.com/paper/1906.08878/) (2019)
- [Bayesian Optimization of Composite Functions](https://mlsanity.com/paper/1906.01537/) (2019)
- [Mixed-Variable Bayesian Optimization](https://mlsanity.com/paper/1907.01329/) (2019)
- [Mercer Features for Efficient Combinatorial Bayesian Optimization](https://mlsanity.com/paper/2012.07762/) (2020)
- [Group kernels for Gaussian process metamodels with categorical inputs](https://mlsanity.com/paper/1802.02368/) (2018)
- [Multi-Fidelity Multi-Objective Bayesian Optimization: An Output Space Entropy Search Approach](https://mlsanity.com/paper/2011.01542/) (2020)
- [Improved Max-value Entropy Search for Multi-objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/2011.01150/) (2020)
- [Optimizing Discrete Spaces via Expensive Evaluations: A Learning to Search Framework](https://mlsanity.com/paper/2012.07320/) (2020)
- [On kernel methods for covariates that are rankings](https://mlsanity.com/paper/1603.08035/) (2016)
- [Scalable Combinatorial Bayesian Optimization with Tractable Statistical models](https://mlsanity.com/paper/2008.08177/) (2020)
- [Learning with Hierarchical Gaussian Kernels](https://mlsanity.com/paper/1612.00824/) (2016)
- [Information-Theoretic Multi-Objective Bayesian Optimization with Continuous Approximations](https://mlsanity.com/paper/2009.05700/) (2020)
- [Uncertainty aware Search Framework for Multi-Objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/2008.07029/) (2020)

## Built on by (most cited first, 23 of 23)

- [Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces](https://mlsanity.com/paper/2111.01186/) (2021)
- [Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization](https://mlsanity.com/paper/2210.10199/) (2022)
- [Sample-Then-Optimize Batch Neural Thompson Sampling](https://mlsanity.com/paper/2210.06850/) (2022)
- [Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddings](https://mlsanity.com/paper/2303.01774/) (2023)
- [A survey and benchmark of high-dimensional Bayesian optimization of discrete sequences](https://mlsanity.com/paper/2406.04739/) (2024)
- [Bayesian optimization as a flexible and efficient design framework for sustainable process systems](https://mlsanity.com/paper/2401.16373/) (2024)
- [Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces](https://mlsanity.com/paper/2207.00879/) (2022)
- [Output Space Entropy Search Framework for Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2110.06980/) (2021)
- [c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization](https://mlsanity.com/paper/2211.14411/) (2022)
- [Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach](https://mlsanity.com/paper/2206.12708/) (2022)
- [On the role of Model Uncertainties in Bayesian Optimization](https://mlsanity.com/paper/2301.05983/) (2023)
- [Optimization on Manifolds via Graph Gaussian Processes](https://mlsanity.com/paper/2210.10962/) (2022)
- [Bayesian Optimization For Multi-Objective Mixed-Variable Problems](https://mlsanity.com/paper/2201.12767/) (2022)
- [Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization](https://mlsanity.com/paper/2206.01409/) (2022)
- [Optimistic Games for Combinatorial Bayesian Optimization with Application to Protein Design](https://mlsanity.com/paper/2409.18582/) (2024)
- [BARK: A Fully Bayesian Tree Kernel for Black-box Optimization](https://mlsanity.com/paper/2503.05574/) (2025)
- [Gearing Gaussian process modeling and sequential design towards stochastic simulators](https://mlsanity.com/paper/2412.07306/) (2024)
- [Discovery of Feasible 3D Printing Configurations for Metal Alloys via AI-driven Adaptive Experimental Design](https://mlsanity.com/paper/2601.17587/) (2026)
- [Mold into a Graph: Efficient Bayesian Optimization over Mixed-Spaces](https://mlsanity.com/paper/2202.00893/) (2022)
- [Non-Myopic Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2412.08085/) (2024)
- [POCAII: Parameter Optimization with Conscious Allocation using Iterative Intelligence](https://mlsanity.com/paper/2505.11745/) (2025)
- [MOCA-HESP: Meta High-dimensional Bayesian Optimization for Combinatorial and Mixed Spaces via Hyper-ellipsoid Partitioning](https://mlsanity.com/paper/2508.06847/) (2025)
- [A reproducible comparative study of categorical kernels for Gaussian process regression, with new clustering-based nested kernels](https://mlsanity.com/paper/2510.01840/) (2025)

## Abstract (arXiv)

We consider the problem of optimizing hybrid structures (mixture of discrete and continuous input variables) via expensive black-box function evaluations. This problem arises in many real-world applications. For example, in materials design optimization via lab experiments, discrete and continuous variables correspond to the presence/absence of primitive elements and their relative concentrations respectively. The key challenge is to accurately model the complex interactions between discrete and continuous variables. In this paper, we propose a novel approach referred as Hybrid Bayesian Optimization (HyBO) by utilizing diffusion kernels, which are naturally defined over continuous and discrete variables. We develop a principled approach for constructing diffusion kernels over hybrid spaces by utilizing the additive kernel formulation, which allows additive interactions of all orders in a tractable manner. We theoretically analyze the modeling strength of additive hybrid kernels and prove that it has the universal approximation property. Our experiments on synthetic and six diverse real-world benchmarks show that HyBO significantly outperforms the state-of-the-art methods.