# Preferential Bayesian Optimization

arXiv: https://arxiv.org/abs/1704.03651 · HTML: https://mlsanity.com/paper/1704.03651/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2017.

## Built on by (most cited first, 30 of 74)

- [Active Preference-Based Gaussian Process Regression for Reward Learning](https://mlsanity.com/paper/2005.02575/) (2020)
- [Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizability](https://mlsanity.com/paper/2111.12306/) (2021)
- [Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes](https://mlsanity.com/paper/2203.11382/) (2022)
- [GAUCHE: A Library for Gaussian Processes in Chemistry](https://mlsanity.com/paper/2212.04450/) (2022)
- [Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Models](https://mlsanity.com/paper/2202.04593/) (2022)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [Looping in the Human Collaborative and Explainable Bayesian Optimization](https://mlsanity.com/paper/2310.17273/) (2023)
- [qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimization](https://mlsanity.com/paper/2303.15746/) (2023)
- [Towards Practical Preferential Bayesian Optimization with Skew Gaussian Processes](https://mlsanity.com/paper/2302.01513/) (2023)
- [Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation](https://mlsanity.com/paper/2405.00981/) (2024)
- [Principled Preferential Bayesian Optimization](https://mlsanity.com/paper/2402.05367/) (2024)
- [Sample-Efficient Alignment for LLMs](https://mlsanity.com/paper/2411.01493/) (2024)
- [Multi-Objective Bayesian Optimization with Active Preference Learning](https://mlsanity.com/paper/2311.13460/) (2023)
- [Exploiting Correlation to Achieve Faster Learning Rates in Low-Rank Preference Bandits](https://mlsanity.com/paper/2202.11795/) (2022)
- [Human-in-the-Loop Optimization for Deep Stimulus Encoding in Visual Prostheses](https://mlsanity.com/paper/2306.13104/) (2023)
- [PABBO: Preferential Amortized Black-Box Optimization](https://mlsanity.com/paper/2503.00924/) (2025)
- [Bandits with Preference Feedback: A Stackelberg Game Perspective](https://mlsanity.com/paper/2406.16745/) (2024)
- [Unleashing LLMs in Bayesian Optimization: Preference-Guided Framework for Scientific Discovery](https://mlsanity.com/paper/2605.17976/) (2026)
- [Toward AI Assistants That Let Designers Design](https://mlsanity.com/paper/2107.13074/) (2021)
- [Explaining Preferences with Shapley Values](https://mlsanity.com/paper/2205.13662/) (2022)
- [Bayesian Optimization Augmented with Actively Elicited Expert Knowledge](https://mlsanity.com/paper/2208.08742/) (2022)
- [Dueling Convex Optimization with General Preferences](https://mlsanity.com/paper/2210.02562/) (2022)
- [Inference-Time Personalized Alignment with a Few User Preference Queries](https://mlsanity.com/paper/2511.02966/) (2025)
- [Efficient Exploration in Binary and Preferential Bayesian Optimization](https://mlsanity.com/paper/2110.09361/) (2021)
- [Principled Bayesian Optimisation in Collaboration with Human Experts](https://mlsanity.com/paper/2410.10452/) (2024)
- [Best Policy Learning from Trajectory Preference Feedback](https://mlsanity.com/paper/2501.18873/) (2025)
- [End-to-End Learning of Deep Kernel Acquisition Functions for Bayesian Optimization](https://mlsanity.com/paper/2111.00639/) (2021)
- [Optimizing Algorithms From Pairwise User Preferences](https://mlsanity.com/paper/2308.04571/) (2023)
- [Ranking over Regression for Bayesian Optimization and Molecule Selection](https://mlsanity.com/paper/2410.09290/) (2024)
- [Streaming Bayes GFlowNets](https://mlsanity.com/paper/2411.05899/) (2024)

## Abstract (arXiv)

Bayesian optimization (BO) has emerged during the last few years as an effective approach to optimizing black-box functions where direct queries of the objective are expensive. In this paper we consider the case where direct access to the function is not possible, but information about user preferences is. Such scenarios arise in problems where human preferences are modeled, such as A/B tests or recommender systems. We present a new framework for this scenario that we call Preferential Bayesian Optimization (PBO) which allows us to find the optimum of a latent function that can only be queried through pairwise comparisons, the so-called duels. PBO extends the applicability of standard BO ideas and generalizes previous discrete dueling approaches by modeling the probability of the winner of each duel by means of a Gaussian process model with a Bernoulli likelihood. The latent preference function is used to define a family of acquisition functions that extend usual policies used in BO. We illustrate the benefits of PBO in a variety of experiments, showing that PBO needs drastically fewer comparisons for finding the optimum. According to our experiments, the way of modeling correlations in PBO is key in obtaining this advantage.