mlsanityData to 7 Oct 2026

On the role of Model Uncertainties in Bayesian Optimization

arXiv 2301.05983 ↗First posted 14 Jan 2023stat.ML cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Bayesian optimization (BO) is a popular method for black-box optimization, which relies on uncertainty as part of its decision-making process when deciding which experiment to perform next. However, not much work has addressed the effect of uncertainty on the performance of the BO algorithm and to what extent calibrated uncertainties improve the ability to find the global optimum. In this work, we provide an extensive study of the relationship between the BO performance (regret) and uncertainty calibration for popular surrogate models and compare them across both synthetic and real-world experiments. Our results confirm that Gaussian Processes are strong surrogate models and that they tend to outperform other popular models. Our results further show a positive association between calibration error and regret, but interestingly, this association disappears when we control for the type of model in the analysis. We also studied the effect of re-calibration and demonstrate that it generally does not lead to improved regret. Finally, we provide theoretical justification for why uncertainty calibration might be difficult to combine with BO due to the small sample sizes commonly used.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 14 Jan 2023.

Builds on

22 of 42 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2016Variational Inference: A Review for Statisticianscited by 1,466
  7. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  8. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  9. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  10. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  11. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
  12. 2019Distribution Calibration for Regressioncited by 88
10 more
  1. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  2. 2013A Literature Survey of Benchmark Functions For Global Optimization Problemscited by 75
  3. 2021Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantificationcited by 57
  4. 2022Uncertainty-Aware Search Framework for Multi-Objective Bayesian Optimizationcited by 43
  5. 2021GradDiv: Adversarial Robustness of Randomized Neural Networks via Gradient Diversity Regularizationcited by 34
  6. 2021Calibrated and Sharp Uncertainties in Deep Learning via Density Estimationcited by 33
  7. 2021Bayesian Optimization over Hybrid Spacescited by 23
  8. 2021Calibrated Uncertainty for Molecular Property Prediction using Ensembles of Message Passing Neural Networkscited by 21
  9. 2021How to Evaluate Uncertainty Estimates in Machine Learning for Regression?cited by 15
  10. 2016A Stratified Analysis of Bayesian Optimization Methodscited by 3

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 4.

7 papers.

  1. 2024Kermut: Composite kernel regression for protein variant effectscited by 11
  2. 2024Ranking over Regression for Bayesian Optimization and Molecule Selectioncited by 4
  3. 2025BioBO: Biology-informed Bayesian Optimization for Perturbation Designcited by 2
  4. 2025Informed Initialization for Bayesian Optimization and Active Learningcited by 2
  5. 2024A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offscited by 1
  6. 2025A physics-informed Bayesian optimization method for rapid development of electrical machinescited by 0
  7. 2025Multi-modal Bayesian Neural Network Surrogates with Conjugate Last-Layer Estimationcited by 0