# Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes

arXiv: https://arxiv.org/abs/2203.11382 · HTML: https://mlsanity.com/paper/2203.11382/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 50 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 19 of 19)

- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2006.05078/) (2020)
- [Maximizing acquisition functions for Bayesian optimization](https://mlsanity.com/paper/1805.10196/) (2018)
- [Constrained Bayesian Optimization with Noisy Experiments](https://mlsanity.com/paper/1706.07094/) (2017)
- [Bayesian optimization for materials design](https://mlsanity.com/paper/1506.01349/) (2015)
- [Preferential Bayesian Optimization](https://mlsanity.com/paper/1704.03651/) (2017)
- [An Easy-to-use Real-world Multi-objective Optimization Problem Suite](https://mlsanity.com/paper/2009.12867/) (2020)
- [Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables](https://mlsanity.com/paper/1910.01688/) (2019)
- [Bayesian Optimization of Composite Functions](https://mlsanity.com/paper/1906.01537/) (2019)
- [Efficient Nonmyopic Bayesian Optimization via One-Shot Multi-Step Trees](https://mlsanity.com/paper/2006.15779/) (2020)
- [Bayesian Optimization of Function Networks](https://mlsanity.com/paper/2112.15311/) (2021)
- [Thinking inside the box: A tutorial on grey-box Bayesian optimization](https://mlsanity.com/paper/2201.00272/) (2022)
- [Multi-Attribute Bayesian Optimization With Interactive Preference Learning](https://mlsanity.com/paper/1911.05934/) (2019)
- [Preferential Batch Bayesian Optimization](https://mlsanity.com/paper/2003.11435/) (2020)
- [Multi-objective Bayesian optimisation with preferences over objectives](https://mlsanity.com/paper/1902.04228/) (2019)
- [Real-world Video Adaptation with Reinforcement Learning](https://mlsanity.com/paper/2008.12858/) (2020)
- [Optimizing High-Dimensional Physics Simulations via Composite Bayesian Optimization](https://mlsanity.com/paper/2111.14911/) (2021)
- [User preferences in Bayesian multi-objective optimization: the expected weighted hypervolume improvement criterion](https://mlsanity.com/paper/1809.05450/) (2018)
- [Interactive Preference Learning of Utility Functions for Multi-Objective Optimization](https://mlsanity.com/paper/1612.04453/) (2016)

## Built on by (most cited first, 30 of 42)

- [Multi-Objective GFlowNets](https://mlsanity.com/paper/2210.12765/) (2022)
- [GAUCHE: A Library for Gaussian Processes in Chemistry](https://mlsanity.com/paper/2212.04450/) (2022)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles](https://mlsanity.com/paper/2303.03751/) (2023)
- [Looping in the Human Collaborative and Explainable Bayesian Optimization](https://mlsanity.com/paper/2310.17273/) (2023)
- [qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimization](https://mlsanity.com/paper/2303.15746/) (2023)
- [Multi-Objective Bayesian Optimization with Active Preference Learning](https://mlsanity.com/paper/2311.13460/) (2023)
- [ComPO: Preference Alignment via Comparison Oracles](https://mlsanity.com/paper/2505.05465/) (2025)
- [PABBO: Preferential Amortized Black-Box Optimization](https://mlsanity.com/paper/2503.00924/) (2025)
- [Collaborative Bayesian Optimization via Wasserstein Barycenters](https://mlsanity.com/paper/2504.10770/) (2025)
- [User Preference Meets Pareto-Optimality in Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2502.06971/) (2025)
- [Joint Composite Latent Space Bayesian Optimization](https://mlsanity.com/paper/2311.02213/) (2023)
- [Preference-Aware Constrained Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2303.13034/) (2023)
- [Optimizing Algorithms From Pairwise User Preferences](https://mlsanity.com/paper/2308.04571/) (2023)
- [Conditioning Gaussian Processes on Almost Anything](https://mlsanity.com/paper/2605.21041/) (2026)
- [Multi-objective optimisation via the R2 utilities](https://mlsanity.com/paper/2305.11774/) (2023)
- [Preferential Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2406.14699/) (2024)
- [Contextual bandits with concave rewards, and an application to fair ranking](https://mlsanity.com/paper/2210.09957/) (2022)
- [Comparative Explanations: Explanation Guided Decision Making for Human-in-the-Loop Preference Selection](https://mlsanity.com/paper/2504.03744/) (2025)
- [Experimenting, Fast and Slow: Bayesian Optimization of Long-term Outcomes with Online Experiments](https://mlsanity.com/paper/2506.18744/) (2025)
- [Bayesian preference elicitation for decision support in multiobjective optimization](https://mlsanity.com/paper/2507.16999/) (2025)
- [The Impact of Expertise in the Loop for Exploring Machine Rationality](https://mlsanity.com/paper/2302.05665/) (2023)
- [Bayesian Optimization with Preference Exploration using a Monotonic Neural Network Ensemble](https://mlsanity.com/paper/2501.18792/) (2025)
- [Exploiting Prior Knowledge in Preferential Learning of Individualized Autonomous Vehicle Driving Styles](https://mlsanity.com/paper/2503.15407/) (2025)
- [An Interactive Framework for Finding the Optimal Trade-off in Differential Privacy](https://mlsanity.com/paper/2509.04290/) (2025)
- [Knowledge Gradient for Preference Learning](https://mlsanity.com/paper/2601.22335/) (2026)
- [SEED-SET: Scalable Evolving Experimental Design for System-level Ethical Testing](https://mlsanity.com/paper/2603.01630/) (2026)
- [Efficient Controller Learning from Human Preferences and Numerical Data Via Multi-Modal Surrogate Models](https://mlsanity.com/paper/2603.24138/) (2026)
- [Preference Elicitation for Policy Optimization and Application to Aligning Heart Transplantation with Human Values](https://mlsanity.com/paper/2608.28620/) (2026)
- [Efficient Contextual Preferential Bayesian Optimization with Historical Examples](https://mlsanity.com/paper/2208.10300/) (2022)

## Abstract (arXiv)

We consider Bayesian optimization of expensive-to-evaluate experiments that generate vector-valued outcomes over which a decision-maker (DM) has preferences. These preferences are encoded by a utility function that is not known in closed form but can be estimated by asking the DM to express preferences over pairs of outcome vectors. To address this problem, we develop Bayesian optimization with preference exploration, a novel framework that alternates between interactive real-time preference learning with the DM via pairwise comparisons between outcomes, and Bayesian optimization with a learned compositional model of DM utility and outcomes. Within this framework, we propose preference exploration strategies specifically designed for this task, and demonstrate their performance via extensive simulation studies.