mlsanityData to 14 Oct 2026

Online Calibrated and Conformal Prediction Improves Bayesian Optimization

arXiv 2112.04620 ↗First posted 8 Dec 2021cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

Accurate uncertainty estimates are important in sequential model-based decision-making tasks such as Bayesian optimization. However, these estimates can be imperfect if the data violates assumptions made by the model (e.g., Gaussianity). This paper studies which uncertainties are needed in model-based decision-making and in Bayesian optimization, and argues that uncertainties can benefit from calibration -- i.e., an 80% predictive interval should contain the true outcome 80% of the time. Maintaining calibration, however, can be challenging when the data is non-stationary and depends on our actions. We propose using simple algorithms based on online learning to provably maintain calibration on non-i.i.d. data, and we show how to integrate these algorithms in Bayesian optimization with minimal overhead. Empirically, we find that calibrated Bayesian optimization converges to better optima in fewer steps, and we demonstrate improved performance on standard benchmark functions and hyperparameter optimization tasks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 17 of its 57 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

17 of 57 references.

References · 17

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2019Introduction to Online Convex Optimizationcited by 803
  4. 2018A Tutorial on Bayesian Optimizationcited by 744
  5. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  6. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 375
  7. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  8. 2019Distribution Calibration for Regressioncited by 89
  9. 2013Structure Discovery in Nonparametric Regression through Compositional Kernel Searchcited by 88
  10. 2015Bayesian optimization for materials designcited by 80
  11. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 76
  12. 2014Input Warping for Bayesian Optimization of Non-stationary Functionscited by 67
5 more
  1. 2019No-Regret Bayesian Optimization with Unknown Hyperparameterscited by 51
  2. 2020Individual Calibration with Randomized Forecastingcited by 50
  3. 2019Calibrated Model-Based Deep Reinforcement Learningcited by 31
  4. 2019Conformal calibratorscited by 22
  5. 2018Learning non-Gaussian Time Series using the Box-Cox Gaussian Processcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 3.

10 papers.

  1. 2024Localized Adaptive Risk Controlcited by 17
  2. 2025Direct Prediction Set Minimization via Bilevel Conformal Classifier Trainingcited by 8
  3. 2024Calibrated Probabilistic Forecasts for Arbitrary Sequencescited by 4
  4. 2024GETS: Ensemble Temperature Scaling for Calibration in Graph Neural Networkscited by 4
  5. 2024Variance-Aware Linear UCB with Deep Representation for Neural Contextual Banditscited by 3
  6. 2024Robust Bayesian Optimization via Localized Online Conformal Predictioncited by 1
  7. 2025Q-Learning with Shift-Aware Upper Confidence Bound in Non-Stationary Reinforcement Learningcited by 1
  8. 2026Online Sharp-Calibrated Bayesian Optimizationcited by 0
  9. 2026Goal-Oriented Lower-Tail Calibration of Gaussian Processes for Bayesian Optimizationcited by 0
  10. 2026Adapting Nonstationary Multi-output Gaussian Processes to Bayesian Optimizationcited by 0