# Knowledge Gradient for Preference Learning

arXiv: https://arxiv.org/abs/2601.22335 · HTML: https://mlsanity.com/paper/2601.22335/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Unexpected Improvements to Expected Improvement for Bayesian Optimization](https://mlsanity.com/paper/2310.20708/) (2023)
- [The Parallel Knowledge Gradient Method for Batch Bayesian Optimization](https://mlsanity.com/paper/1606.04414/) (2016)
- [Preferential Bayesian Optimization](https://mlsanity.com/paper/1704.03651/) (2017)
- [Bayesian Optimization with Gradients](https://mlsanity.com/paper/1703.04389/) (2017)
- [Convergence of Sparse Variational Inference in Gaussian Processes Regression](https://mlsanity.com/paper/2008.00323/) (2020)
- [Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomes](https://mlsanity.com/paper/2203.11382/) (2022)
- [qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimization](https://mlsanity.com/paper/2303.15746/) (2023)
- [Projective Preferential Bayesian Optimization](https://mlsanity.com/paper/2002.03113/) (2020)
- [ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapes](https://mlsanity.com/paper/2011.04812/) (2020)
- [Towards Practical Preferential Bayesian Optimization with Skew Gaussian Processes](https://mlsanity.com/paper/2302.01513/) (2023)
- [Preferential Bayesian optimisation with Skew Gaussian Processes](https://mlsanity.com/paper/2008.06677/) (2020)
- [Principled Preferential Bayesian Optimization](https://mlsanity.com/paper/2402.05367/) (2024)
- [Integrals over Gaussians under Linear Domain Constraints](https://mlsanity.com/paper/1910.09328/) (2019)
- [Skew Gaussian Processes for Classification](https://mlsanity.com/paper/2005.12987/) (2020)
- [Look-Ahead Acquisition Functions for Bernoulli Level Set Estimation](https://mlsanity.com/paper/2203.09751/) (2022)
- [Mixed Likelihood Variational Gaussian Processes](https://mlsanity.com/paper/2503.04138/) (2025)
- [A Fast, Robust Elliptical Slice Sampling Implementation for Linearly Truncated Multivariate Normal Distributions](https://mlsanity.com/paper/2407.10449/) (2024)

## Built on by (most cited first, 1 of 1)

- [Elicitation-Augmented Bayesian Optimization](https://mlsanity.com/paper/2605.12079/) (2026)

## Abstract (arXiv)

The knowledge gradient is a popular acquisition function in Bayesian optimization (BO) for optimizing black-box objectives with noisy function evaluations. Many practical settings, however, allow only pairwise comparison queries, yielding a preferential BO problem where direct function evaluations are unavailable. Extending the knowledge gradient to preferential BO is hindered by its computational challenge. At its core, the look-ahead step in the preferential setting requires computing a non-Gaussian posterior, which was previously considered intractable. In this paper, we address this challenge by deriving an exact and analytical knowledge gradient for preferential BO. We show that the exact knowledge gradient performs strongly on a suite of benchmark problems, often outperforming existing acquisition functions. In addition, we also present a case study illustrating the limitation of the knowledge gradient in certain scenarios.