mlsanityData to 7 Oct 2026

Knowledge Gradient for Preference Learning

arXiv 2601.22335 ↗First posted 29 Jan 2026cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

The knowledge gradient is a popular acquisition function in Bayesian optimization (BO) for optimizing black-box objectives with noisy function evaluations. Many practical settings, however, allow only pairwise comparison queries, yielding a preferential BO problem where direct function evaluations are unavailable. Extending the knowledge gradient to preferential BO is hindered by its computational challenge. At its core, the look-ahead step in the preferential setting requires computing a non-Gaussian posterior, which was previously considered intractable. In this paper, we address this challenge by deriving an exact and analytical knowledge gradient for preferential BO. We show that the exact knowledge gradient performs strongly on a suite of benchmark problems, often outperforming existing acquisition functions. In addition, we also present a case study illustrating the limitation of the knowledge gradient in certain scenarios.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

23 of 39 references.

References · 23

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2018A Tutorial on Bayesian Optimizationcited by 734
  6. 2014Scalable Variational Gaussian Process Classificationcited by 226
  7. 2023Unexpected Improvements to Expected Improvement for Bayesian Optimizationcited by 144
  8. 2016The Parallel Knowledge Gradient Method for Batch Bayesian Optimizationcited by 73
  9. 2017Preferential Bayesian Optimizationcited by 72
  10. 2017Bayesian Optimization with Gradientscited by 59
  11. 2020Convergence of Sparse Variational Inference in Gaussian Processes Regressioncited by 53
  12. 2022Preference Exploration for Efficient Bayesian Optimization with Multiple Outcomescited by 41
11 more
  1. 2023qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimizationcited by 23
  2. 2020Projective Preferential Bayesian Optimizationcited by 21
  3. 2020ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapescited by 19
  4. 2023Towards Practical Preferential Bayesian Optimization with Skew Gaussian Processescited by 19
  5. 2020Preferential Bayesian optimisation with Skew Gaussian Processescited by 18
  6. 2024Principled Preferential Bayesian Optimizationcited by 16
  7. 2019Integrals over Gaussians under Linear Domain Constraintscited by 15
  8. 2020Skew Gaussian Processes for Classificationcited by 8
  9. 2022Look-Ahead Acquisition Functions for Bernoulli Level Set Estimationcited by 6
  10. 2025Mixed Likelihood Variational Gaussian Processescited by 2
  11. 2024A Fast, Robust Elliptical Slice Sampling Implementation for Linearly Truncated Multivariate Normal Distributionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Elicitation-Augmented Bayesian Optimizationcited by 0