# Bayesian Optimization with Conformal Prediction Sets

arXiv: https://arxiv.org/abs/2210.12496 · HTML: https://mlsanity.com/paper/2210.12496/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 69 references that are in the map.
- First posted on arXiv 22 Oct 2022.

## Builds on (most cited first, 30 of 32)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2006.05078/) (2020)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Learning with Good Feature Representations in Bandits and in RL with a Generative Model](https://mlsanity.com/paper/1911.07676/) (2019)
- [Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement](https://mlsanity.com/paper/2105.08195/) (2021)
- [Learning Optimal Conformal Classifiers](https://mlsanity.com/paper/2110.09192/) (2021)
- [Constrained Bayesian Optimization with Noisy Experiments](https://mlsanity.com/paper/1706.07094/) (2017)
- [Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders](https://mlsanity.com/paper/2203.12742/) (2022)
- [Multi-Objective Bayesian Optimization over High-Dimensional Search Spaces](https://mlsanity.com/paper/2109.10964/) (2021)
- [Adapting to Misspecification in Contextual Bandits](https://mlsanity.com/paper/2107.05745/) (2021)
- [The reparameterization trick for acquisition functions](https://mlsanity.com/paper/1712.00424/) (2017)
- [Misspecified Gaussian Process Bandit Optimization](https://mlsanity.com/paper/2111.05008/) (2021)
- [Conformal Bayesian Computation](https://mlsanity.com/paper/2106.06137/) (2021)
- [Robust Multi-Objective Bayesian Optimization Under Input Noise](https://mlsanity.com/paper/2202.07549/) (2022)
- [Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior](https://mlsanity.com/paper/1811.09558/) (2018)
- [Efficient and Differentiable Conformal Prediction with General Function Classes](https://mlsanity.com/paper/2202.11091/) (2022)
- [Conditioning Sparse Variational Gaussian Processes for Online Decision-making](https://mlsanity.com/paper/2110.15172/) (2021)
- [Bayes-optimal prediction with frequentist coverage control](https://mlsanity.com/paper/2105.14045/) (2021)
- [Stable Conformal Prediction Sets](https://mlsanity.com/paper/2112.10224/) (2021)
- [Kernel Interpolation for Scalable Online Gaussian Processes](https://mlsanity.com/paper/2103.01454/) (2021)
- [Risk-averse Heteroscedastic Bayesian Optimization](https://mlsanity.com/paper/2111.03637/) (2021)
- [Modular Conformal Calibration](https://mlsanity.com/paper/2206.11468/) (2022)

## Built on by (most cited first, 28 of 28)

- [Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)](https://mlsanity.com/paper/2405.06627/) (2024)
- [Self-Correcting Bayesian Optimization through Bayesian Active Learning](https://mlsanity.com/paper/2304.11005/) (2023)
- [Label-wise Aleatoric and Epistemic Uncertainty Quantification](https://mlsanity.com/paper/2406.02354/) (2024)
- [Second-Order Uncertainty Quantification: Variance-Based Measures](https://mlsanity.com/paper/2401.00276/) (2023)
- [WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales](https://mlsanity.com/paper/2505.04608/) (2025)
- [Bayesian Optimization with Formal Safety Guarantees via Online Conformal Prediction](https://mlsanity.com/paper/2306.17815/) (2023)
- [Conformalized Interval Arithmetic with Symmetric Calibration](https://mlsanity.com/paper/2408.10939/) (2024)
- [Conformal Approach To Gaussian Process Surrogate Evaluation With Coverage Guarantees](https://mlsanity.com/paper/2401.07733/) (2024)
- [Conformal Prediction as Bayesian Quadrature](https://mlsanity.com/paper/2502.13228/) (2025)
- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [Conformalized Deep Splines for Optimal and Efficient Prediction Sets](https://mlsanity.com/paper/2311.00774/) (2023)
- [Risk-Controlling Model Selection via Guided Bayesian Optimization](https://mlsanity.com/paper/2312.01692/) (2023)
- [CUQDS: Conformal Uncertainty Quantification under Distribution Shift for Trajectory Prediction](https://mlsanity.com/paper/2406.12100/) (2024)
- [Beyond the training set: an intuitive method for detecting distribution shift in model-based optimization](https://mlsanity.com/paper/2311.05363/) (2023)
- [Efficient Online Set-valued Classification with Bandit Feedback](https://mlsanity.com/paper/2405.04393/) (2024)
- [Online scalable Gaussian processes with conformal prediction for guaranteed coverage](https://mlsanity.com/paper/2410.05444/) (2024)
- [Is novelty predictable?](https://mlsanity.com/paper/2306.00872/) (2023)
- [Smart Surrogate Losses for Contextual Stochastic Linear Optimization with Robust Constraints](https://mlsanity.com/paper/2505.22881/) (2025)
- [Anchor-Based Heteroscedastic Noise for Preferential Bayesian Optimization](https://mlsanity.com/paper/2405.14657/) (2024)
- [Robust Bayesian Optimization via Localized Online Conformal Prediction](https://mlsanity.com/paper/2411.17387/) (2024)
- [On the Out-of-Distribution Coverage of Combining Split Conformal Prediction and Bayesian Deep Learning](https://mlsanity.com/paper/2311.12688/) (2023)
- [Aerial Image Classification in Scarce and Unconstrained Environments via Conformal Prediction](https://mlsanity.com/paper/2504.17655/) (2025)
- [Conformal bandits: bringing statistical validity and reward efficiency under weak arm separability](https://mlsanity.com/paper/2512.09850/) (2025)
- [Online Sharp-Calibrated Bayesian Optimization](https://mlsanity.com/paper/2605.10572/) (2026)
- [Goal-Oriented Lower-Tail Calibration of Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2605.20145/) (2026)
- [AK-MCS-C2 : Active Kriging Monte Carlo Simulation method with conformal certification for failure probability estimation](https://mlsanity.com/paper/2606.20191/) (2026)
- [Adapting Nonstationary Multi-output Gaussian Processes to Bayesian Optimization](https://mlsanity.com/paper/2609.32464/) (2026)
- [Coverage You Can Steer: Online Conformal Calibration for RL-Driven Hardware-Aware NAS](https://mlsanity.com/paper/2610.03127/) (2026)

## Abstract (arXiv)

Bayesian optimization is a coherent, ubiquitous approach to decision-making under uncertainty, with applications including multi-arm bandits, active learning, and black-box optimization. Bayesian optimization selects decisions (i.e. objective function queries) with maximal expected utility with respect to the posterior distribution of a Bayesian model, which quantifies reducible, epistemic uncertainty about query outcomes. In practice, subjectively implausible outcomes can occur regularly for two reasons: 1) model misspecification and 2) covariate shift. Conformal prediction is an uncertainty quantification method with coverage guarantees even for misspecified models and a simple mechanism to correct for covariate shift. We propose conformal Bayesian optimization, which directs queries towards regions of search space where the model predictions have guaranteed validity, and investigate its behavior on a suite of black-box optimization tasks and tabular ranking tasks. In many cases we find that query coverage can be significantly improved without harming sample-efficiency.