# Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization

arXiv: https://arxiv.org/abs/2306.09803 · HTML: https://mlsanity.com/paper/2306.09803/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 106 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [GPflow: A Gaussian process library using TensorFlow](https://mlsanity.com/paper/1610.08733/) (2016)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [Bayesian Neural Networks: An Introduction and Survey](https://mlsanity.com/paper/2006.12024/) (2020)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [Batch Bayesian Optimization via Local Penalization](https://mlsanity.com/paper/1505.08052/) (2015)
- [Combinatorial Bayesian Optimization using the Graph Cartesian Product](https://mlsanity.com/paper/1902.00448/) (2019)
- [HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML](https://mlsanity.com/paper/2106.06257/) (2021)
- [The Parallel Knowledge Gradient Method for Batch Bayesian Optimization](https://mlsanity.com/paper/1606.04414/) (2016)
- [Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly](https://mlsanity.com/paper/1903.06694/) (2019)
- [Bayesian Optimization of Combinatorial Structures](https://mlsanity.com/paper/1806.08838/) (2018)
- [YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization](https://mlsanity.com/paper/2109.03670/) (2021)
- [Bayesian Optimisation over Multiple Continuous and Categorical Inputs](https://mlsanity.com/paper/1906.08878/) (2019)
- [BOSS: Bayesian Optimization over String Spaces](https://mlsanity.com/paper/2010.00979/) (2020)
- [Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces](https://mlsanity.com/paper/2102.07188/) (2021)
- [Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization](https://mlsanity.com/paper/2210.10199/) (2022)
- [Emulation of physical processes with Emukit](https://mlsanity.com/paper/2110.13293/) (2021)
- [Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddings](https://mlsanity.com/paper/2303.01774/) (2023)
- [GPflowOpt: A Bayesian Optimization Library using TensorFlow](https://mlsanity.com/paper/1711.03845/) (2017)
- [BOiLS: Bayesian Optimisation for Logic Synthesis](https://mlsanity.com/paper/2111.06178/) (2021)
- [AntBO: Towards Real-World Automated Antibody Design with Combinatorial Bayesian Optimisation](https://mlsanity.com/paper/2201.12570/) (2022)
- [Are we Forgetting about Compositional Optimisers in Bayesian Optimisation?](https://mlsanity.com/paper/2012.08240/) (2020)
- [The EPFL Logic Synthesis Libraries](https://mlsanity.com/paper/1805.05121/) (2018)

## Built on by (most cited first, 16 of 16)

- [A survey and benchmark of high-dimensional Bayesian optimization of discrete sequences](https://mlsanity.com/paper/2406.04739/) (2024)
- [Bayesian optimization as a flexible and efficient design framework for sustainable process systems](https://mlsanity.com/paper/2401.16373/) (2024)
- [Surprisingly Strong Performance Prediction with Neural Graph Features](https://mlsanity.com/paper/2404.16551/) (2024)
- [Optimistic Games for Combinatorial Bayesian Optimization with Application to Protein Design](https://mlsanity.com/paper/2409.18582/) (2024)
- [BARK: A Fully Bayesian Tree Kernel for Black-box Optimization](https://mlsanity.com/paper/2503.05574/) (2025)
- [CATBench: A Compiler Autotuning Benchmarking Suite for Black-box Optimization](https://mlsanity.com/paper/2406.17811/) (2024)
- [Fine-tuning LLMs with variational Bayesian last layer for high-dimensional Bayesian optimization](https://mlsanity.com/paper/2510.01471/) (2025)
- [Evaluating Useful Surrogate Models for Configuration Tuning Beyond Accuracy: A Fitness Landscape Analysis Perspective](https://mlsanity.com/paper/2509.21945/) (2025)
- [Omnipresent Yet Overlooked: Heat Kernels in Combinatorial Bayesian Optimization](https://mlsanity.com/paper/2510.26633/) (2025)
- [BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks](https://mlsanity.com/paper/2605.17000/) (2026)
- [Weighted Euclidean Distance Matrices over Mixed Continuous and Categorical Inputs for Gaussian Process Models](https://mlsanity.com/paper/2503.02630/) (2025)
- [Tree-Structured Parzen Estimator Can Solve Black-Box Combinatorial Optimization More Efficiently](https://mlsanity.com/paper/2507.08053/) (2025)
- [MOCA-HESP: Meta High-dimensional Bayesian Optimization for Combinatorial and Mixed Spaces via Hyper-ellipsoid Partitioning](https://mlsanity.com/paper/2508.06847/) (2025)
- [COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier](https://mlsanity.com/paper/2510.01178/) (2025)
- [A reproducible comparative study of categorical kernels for Gaussian process regression, with new clustering-based nested kernels](https://mlsanity.com/paper/2510.01840/) (2025)
- [Bayesian Optimization for Mixed-Variable Problems in the Natural Sciences](https://mlsanity.com/paper/2604.07416/) (2026)

## Abstract (arXiv)

This paper introduces a modular framework for Mixed-variable and Combinatorial Bayesian Optimization (MCBO) to address the lack of systematic benchmarking and standardized evaluation in the field. Current MCBO papers often introduce non-diverse or non-standard benchmarks to evaluate their methods, impeding the proper assessment of different MCBO primitives and their combinations. Additionally, papers introducing a solution for a single MCBO primitive often omit benchmarking against baselines that utilize the same methods for the remaining primitives. This omission is primarily due to the significant implementation overhead involved, resulting in a lack of controlled assessments and an inability to showcase the merits of a contribution effectively. To overcome these challenges, our proposed framework enables an effortless combination of Bayesian Optimization components, and provides a diverse set of synthetic and real-world benchmarking tasks. Leveraging this flexibility, we implement 47 novel MCBO algorithms and benchmark them against seven existing MCBO solvers and five standard black-box optimization algorithms on ten tasks, conducting over 4000 experiments. Our findings reveal a superior combination of MCBO primitives outperforming existing approaches and illustrate the significance of model fit and the use of a trust region. We make our MCBO library available under the MIT license at urlhttps://github.com/huawei-noah/HEBO/tree/master/MCBO.