mlsanityData to 7 Oct 2026

Principled Preferential Bayesian Optimization

arXiv 2402.05367 ↗First posted 8 Feb 2024cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We study the problem of preferential Bayesian optimization (BO), where we aim to optimize a black-box function with only preference feedback over a pair of candidate solutions. Inspired by the likelihood ratio idea, we construct a confidence set of the black-box function using only the preference feedback. An optimistic algorithm with an efficient computational method is then developed to solve the problem, which enjoys an information-theoretic bound on the total cumulative regret, a first-of-its-kind for preferential BO. This bound further allows us to design a scheme to report an estimated best solution, with a guaranteed convergence rate. Experimental results on sampled instances from Gaussian processes, standard test functions, and a thermal comfort optimization problem all show that our method stably achieves better or competitive performance as compared to the existing state-of-the-art heuristics, which, however, do not have theoretical guarantees on regret bounds or convergence.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 33 of its 65 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

33 of 65 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2018A Tutorial on Bayesian Optimizationcited by 734
  4. 2017On Kernelized Multi-armed Banditscited by 348
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  6. 2015High Dimensional Bayesian Optimisation and Bandits via Additive Modelscited by 137
  7. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  8. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  9. 2015Contextual Dueling Banditscited by 101
  10. 2023Is RLHF More Difficult than Standard RL?cited by 96
  11. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  12. 2018Stagewise Safe Bayesian Optimization with Gaussian Processescited by 79
21 more
  1. 2015Safe Controller Optimization for Quadrotors with Gaussian Processescited by 77
  2. 2017Preferential Bayesian Optimizationcited by 72
  3. 2022Constrained Efficient Global Optimization of Expensive Black-box Functionscited by 49
  4. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  5. 2017Multi-dueling Bandits with Dependent Armscited by 35
  6. 2016A Relative Exponential Weighing Algorithm for Adversarial Utility-based Dueling Banditscited by 31
  7. 2020Sequential Gallery for Interactive Visual Design Optimizationcited by 24
  8. 2020Deterministic error bounds for kernel-based learning techniques under bounded noisecited by 24
  9. 2022On Kernelized Multi-Armed Bandits with Constraintscited by 23
  10. 2023qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimizationcited by 23
  11. 2020Projective Preferential Bayesian Optimizationcited by 21
  12. 2020ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapescited by 19
  13. 2023Towards Practical Preferential Bayesian Optimization with Skew Gaussian Processescited by 19
  14. 2023Primitive Skill-based Robot Learning from Human Evaluative Feedbackcited by 15
  15. 2020Towards Tractable Optimism in Model-Based Reinforcement Learningcited by 10
  16. 2021VABO: Violation-Aware Bayesian Optimization for Closed-Loop Control Performance Optimization with Unmodeled Constraintscited by 10
  17. 2023Likelihood Ratio Confidence Sets for Sequential Decision Makingcited by 10
  18. 2022Lower Bounds on the Worst-Case Complexity of Efficient Global Optimizationcited by 7
  19. 2023Kernelized Offline Contextual Dueling Banditscited by 4
  20. 2022Targeting occupant feedback using digital twins: Adaptive spatial-temporal thermal preference sampling to optimize personal comfort modelscited by 3
  21. 2022Safe Zeroth-Order Convex Optimization Using Quadratic Local Approximationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 7.

16 papers.

Sort:

By area:

  1. 2024Optimal Design for Human Preference Elicitationcited by 17
  2. 2024Bandits with Preference Feedback: A Stackelberg Game Perspectivecited by 7
  3. 2025PABBO: Preferential Amortized Black-Box Optimizationcited by 7
  4. 2024Principled Bayesian Optimisation in Collaboration with Human Expertscited by 5
  5. 2025Bayesian Optimization from Human Feedback: Near-Optimal Regret Boundscited by 3
  6. 2024Learning from Imperfect Human Feedback: a Tale from Corruption-Robust Duelingcited by 2
  7. 2024Anchor-Based Heteroscedastic Noise for Preferential Bayesian Optimizationcited by 1
  8. 2025Bayesian Optimization with Preference Exploration using a Monotonic Neural Network Ensemblecited by 1
  9. 2026Human-LLM Collaborative Feature Engineering for Tabular Datacited by 1
  10. 2026Knowledge Gradient for Preference Learningcited by 1
  11. 2025Consecutive Preferential Bayesian Optimizationcited by 0
  12. 2026A Finite Time Analysis of Thompson Sampling for Bayesian Optimization with Preferential Feedbackcited by 0
4 more
  1. 2026Elicitation-Augmented Bayesian Optimizationcited by 0
  2. 2026Local Preferential Bayesian Optimizationcited by 0
  3. 2026Adaptive KappaSharp: Condition-Number Shaping for Preferential Bayesian Optimizationcited by 0
  4. 2026On the Complexity of Preference-Based Banditscited by 0