# Preferential Multi-Objective Bayesian Optimization

arXiv: https://arxiv.org/abs/2406.14699 · HTML: https://mlsanity.com/paper/2406.14699/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2006.05078/) (2020)
- [Preference-based Online Learning with Dueling Bandits: A Survey](https://mlsanity.com/paper/1807.11398/) (2018)
- [A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalarizations](https://mlsanity.com/paper/1805.12168/) (2018)
- [Max-value Entropy Search for Multi-Objective Bayesian Optimization with Constraints](https://mlsanity.com/paper/2009.01721/) (2020)
- [Preferential Bayesian Optimization](https://mlsanity.com/paper/1704.03651/) (2017)
- [An Easy-to-use Real-world Multi-objective Optimization Problem Suite](https://mlsanity.com/paper/2009.12867/) (2020)
- [Asking Easy Questions: A User-Friendly Approach to Active Reward Learning](https://mlsanity.com/paper/1910.04365/) (2019)
- [Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables](https://mlsanity.com/paper/1910.01688/) (2019)
- [Dueling Posterior Sampling for Preference-Based Reinforcement Learning](https://mlsanity.com/paper/1908.01289/) (2019)
- [Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes](https://mlsanity.com/paper/2203.11382/) (2022)
- [Multi-dueling Bandits with Dependent Arms](https://mlsanity.com/paper/1705.00253/) (2017)
- [Preference-Based Learning for Exoskeleton Gait Optimization](https://mlsanity.com/paper/1909.12316/) (2019)
- [qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimization](https://mlsanity.com/paper/2303.15746/) (2023)
- [Top-k Ranking Bayesian Optimization](https://mlsanity.com/paper/2012.10688/) (2020)
- [Bayesian Optimization for Policy Search via Online-Offline Experimentation](https://mlsanity.com/paper/1904.01049/) (2019)
- [Preferential Batch Bayesian Optimization](https://mlsanity.com/paper/2003.11435/) (2020)
- [Towards Practical Preferential Bayesian Optimization with Skew Gaussian Processes](https://mlsanity.com/paper/2302.01513/) (2023)
- [Preference learning along multiple criteria: A game-theoretic perspective](https://mlsanity.com/paper/2105.01850/) (2021)
- [Human Preference-Based Learning for High-dimensional Optimization of Exoskeleton Walking Gaits](https://mlsanity.com/paper/2003.06495/) (2020)
- [Preference-Based Learning for User-Guided HZD Gait Generation on Bipedal Walking Robots](https://mlsanity.com/paper/2011.05424/) (2020)
- [Learning Controller Gains on Bipedal Walking Robots via User Preferences](https://mlsanity.com/paper/2102.13201/) (2021)

## Built on by (most cited first, 3 of 3)

- [PABBO: Preferential Amortized Black-Box Optimization](https://mlsanity.com/paper/2503.00924/) (2025)
- [Bayesian Optimization with Preference Exploration using a Monotonic Neural Network Ensemble](https://mlsanity.com/paper/2501.18792/) (2025)
- [Human-in-the-Loop Pareto Optimization: Trade-off Characterization for Assist-as-Needed Training and Performance Evaluation](https://mlsanity.com/paper/2603.23777/) (2026)

## Abstract (arXiv)

Preferential Bayesian optimization (PBO) is a framework for optimizing a decision-maker's latent preferences over available design choices. While preferences often involve multiple conflicting objectives, existing work in PBO assumes that preferences can be encoded by a single objective function. For example, in robotic assistive devices, technicians often attempt to maximize user comfort while simultaneously minimizing mechanical energy consumption for longer battery life. Similarly, in autonomous driving policy design, decision-makers wish to understand the trade-offs between multiple safety and performance attributes before committing to a policy. To address this gap, we propose the first framework for PBO with multiple objectives. Within this framework, we present dueling scalarized Thompson sampling (DSTS), a multi-objective generalization of the popular dueling Thompson algorithm, which may be of interest beyond the PBO setting. We evaluate DSTS across four synthetic test functions and two simulated exoskeleton personalization and driving policy design tasks, showing that it outperforms several benchmarks. Finally, we prove that DSTS is asymptotically consistent. As a direct consequence, this result provides, to our knowledge, the first convergence guarantee for dueling Thompson sampling in the PBO setting.