mlsanityData to 7 Oct 2026

Risk-Controlling Model Selection via Guided Bayesian Optimization

arXiv 2312.01692 ↗First posted 4 Dec 2023cs.LG cs.AI stat.MEcited by 5

Not on the map. Why?

Abstract (arXiv)

Adjustable hyperparameters of machine learning models typically impact various key trade-offs such as accuracy, fairness, robustness, or inference cost. Our goal in this paper is to find a configuration that adheres to user-specified limits on certain risks while being useful with respect to other conflicting metrics. We solve this by combining Bayesian Optimization (BO) with rigorous risk-controlling procedures, where our core idea is to steer BO towards an efficient testing strategy. Our BO method identifies a set of Pareto optimal configurations residing in a designated region of interest. The resulting candidates are statistically verified and the best-performing configuration is selected with guaranteed risk levels. We demonstrate the effectiveness of our approach on a range of tasks with multiple desiderata, including low error rates, equitable predictions, handling spurious correlations, managing rate and distortion in generative models, and reducing computational costs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 40 of its 73 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

40 of 73 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  5. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  6. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  7. 2018A Tutorial on Bayesian Optimizationcited by 734
  8. 2016Distribution-Free Predictive Inference For Regressioncited by 657
  9. 2022Conformal Risk Controlcited by 290
  10. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  11. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  12. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
28 more
  1. 2019Pareto Multi-Task Learningcited by 224
  2. 2019Predictive inference with the jackknife+cited by 206
  3. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  4. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  5. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  6. 2020Learning the Pareto Front with Hypernetworkscited by 122
  7. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 119
  8. 2021Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvementcited by 115
  9. 2017Constrained Bayesian Optimization with Noisy Experimentscited by 103
  10. 2018A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalarizationscited by 98
  11. 2015Predictive Entropy Search for Multi-objective Bayesian Optimizationcited by 88
  12. 2020Max-value Entropy Search for Multi-Objective Bayesian Optimization with Constraintscited by 81
  13. 2020Distribution-free binary classification: prediction sets, confidence intervals and calibrationcited by 79
  14. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 74
  15. 2022Recent Advances in Bayesian Optimizationcited by 74
  16. 2020Controllable Pareto Multi-Task Learningcited by 51
  17. 2021Scalable Pareto Front Approximation for Deep Multi-Objective Learningcited by 45
  18. 2022Uncertainty-Aware Search Framework for Multi-Objective Bayesian Optimizationcited by 43
  19. 2021Zero Time Waste: Recycling Predictions in Early Exit Neural Networkscited by 39
  20. 2022Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overviewcited by 37
  21. 2022Bayesian Optimization with Conformal Prediction Setscited by 27
  22. 2022Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Casecited by 26
  23. 2020Addressing Fairness in Classification with a Model-Agnostic Multi-Objective Algorithmcited by 23
  24. 2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 21
  25. 2015Large-scale probabilistic predictors with and without guarantees of validitycited by 18
  26. 2023Optimizing Hyperparameters with Conformal Quantile Regressioncited by 10
  27. 2023Bayesian Optimization with Formal Safety Guarantees via Online Conformal Predictioncited by 10
  28. 2022Mind the Gap: Measuring Generalization Performance Across Multiple Objectivescited by 3

Built on by

Cited by year

Citing papers per year: 2024: 5.

5 papers.

  1. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  2. 2024Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)cited by 26
  3. 2024Semi-Supervised Risk Control via Prediction-Powered Inferencecited by 13
  4. 2024Early Time Classification with Accumulated Accuracy Gap Controlcited by 11
  5. 2024Confident magnitude-based neural network pruningcited by 1