# Bayesian Optimization for Categorical and Category-Specific Continuous Inputs

arXiv: https://arxiv.org/abs/1911.12473 · HTML: https://mlsanity.com/paper/1911.12473/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2019.

## Built on by (most cited first, 28 of 28)

- [Recent Advances in Bayesian Optimization](https://mlsanity.com/paper/2206.03301/) (2022)
- [Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces](https://mlsanity.com/paper/2102.07188/) (2021)
- [Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces](https://mlsanity.com/paper/2111.01186/) (2021)
- [Joint Entropy Search for Multi-objective Bayesian Optimization](https://mlsanity.com/paper/2210.02905/) (2022)
- [Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterization](https://mlsanity.com/paper/2210.10199/) (2022)
- [Surrogate modeling for Bayesian optimization beyond a single Gaussian process](https://mlsanity.com/paper/2205.14090/) (2022)
- [A mixed-categorical correlation kernel for Gaussian process](https://mlsanity.com/paper/2211.08262/) (2022)
- [Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces](https://mlsanity.com/paper/2207.00879/) (2022)
- [Generating Realistic Tabular Data with Large Language Models](https://mlsanity.com/paper/2410.21717/) (2024)
- [Bayesian optimization for mixed variables using an adaptive dimension reduction process: applications to aircraft design](https://mlsanity.com/paper/2504.08682/) (2025)
- [Bayesian Optimization For Multi-Objective Mixed-Variable Problems](https://mlsanity.com/paper/2201.12767/) (2022)
- [Mixed Variable Bayesian Optimization with Frequency Modulated Kernels](https://mlsanity.com/paper/2102.12792/) (2021)
- [Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization](https://mlsanity.com/paper/2206.01409/) (2022)
- [Mastering the exploration-exploitation trade-off in Bayesian Optimization](https://mlsanity.com/paper/2305.08624/) (2023)
- [Practical First-Order Bayesian Optimization Algorithms](https://mlsanity.com/paper/2306.10815/) (2023)
- [Explaining Inference Queries with Bayesian Optimization](https://mlsanity.com/paper/2102.05308/) (2021)
- [Efficient Classification with Counterfactual Reasoning and Active Learning](https://mlsanity.com/paper/2207.12086/) (2022)
- [Co-Design of Out-of-Distribution Detectors for Autonomous Emergency Braking Systems](https://mlsanity.com/paper/2307.13419/) (2023)
- [3D Reconstruction in Noisy Agricultural Environments: A Bayesian Optimization Perspective for View Planning](https://mlsanity.com/paper/2310.00145/) (2023)
- [From Sorting Algorithms to Scalable Kernels: Bayesian Optimization in High-Dimensional Permutation Spaces](https://mlsanity.com/paper/2507.13263/) (2025)
- [Bayesian Optimisation for Mixed-Variable Inputs using Value Proposals](https://mlsanity.com/paper/2202.04832/) (2022)
- [Evaluating Automated Driving Planner Robustness against Adversarial Influence](https://mlsanity.com/paper/2205.14697/) (2022)
- [A Unified Gaussian Process for Branching and Nested Hyperparameter Optimization](https://mlsanity.com/paper/2402.04885/) (2024)
- [Modeling All Response Surfaces in One for Conditional Search Spaces](https://mlsanity.com/paper/2501.04260/) (2025)
- [Bayesian Optimization for Simultaneous Selection of Machine Learning Algorithms and Hyperparameters on Shared Latent Space](https://mlsanity.com/paper/2502.09329/) (2025)
- [A reproducible comparative study of categorical kernels for Gaussian process regression, with new clustering-based nested kernels](https://mlsanity.com/paper/2510.01840/) (2025)
- [Combinatorial Bandit Bayesian Optimization for Tensor Outputs](https://mlsanity.com/paper/2602.00640/) (2026)
- [SMOTE-VAR: An Uncertainty-Aware Oversampling Method for Predicting Depression Remission in University Students](https://mlsanity.com/paper/2608.30102/) (2026)

## Abstract (arXiv)

Many real-world functions are defined over both categorical and category-specific continuous variables and thus cannot be optimized by traditional Bayesian optimization (BO) methods. To optimize such functions, we propose a new method that formulates the problem as a multi-armed bandit problem, wherein each category corresponds to an arm with its reward distribution centered around the optimum of the objective function in continuous variables. Our goal is to identify the best arm and the maximizer of the corresponding continuous function simultaneously. Our algorithm uses a Thompson sampling scheme that helps connecting both multi-arm bandit and BO in a unified framework. We extend our method to batch BO to allow parallel optimization when multiple resources are available. We theoretically analyze our method for convergence and prove sub-linear regret bounds. We perform a variety of experiments: optimization of several benchmark functions, hyper-parameter tuning of a neural network, and automatic selection of the best machine learning model along with its optimal hyper-parameters (a.k.a automated machine learning). Comparisons with other methods demonstrate the effectiveness of our proposed method.