mlsanityData to 14 Oct 2026

A Finite Time Analysis of Thompson Sampling for Bayesian Optimization with Preferential Feedback

arXiv 2604.25025 ↗First posted 27 Apr 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Preference feedback, in the form of pairwise comparisons rather than scalar scores, has seen increasing use in applications such as human-, laboratory-, and expert-in-the-loop design, as well as scientific discovery. We propose a Thompson Sampling (TS) approach to Bayesian optimization with preferential feedback that models comparisons using a monotone link on latent utility differences and leverages the dueling kernel induced by a base kernel. We provide a finite-time analysis showing that the performance of the proposed method matches that of standard TS for conventional Bayesian optimization with scalar feedback. The analysis exploits the anchor invariance of TS for challenger selection and introduces a double-TS pairing variant. We also demonstrate the performance of the method on both synthetic and real-world examples.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 45 of its 73 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

45 of 73 references.

References · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  7. 2017On Kernelized Multi-armed Banditscited by 352
  8. 2013Learning to Optimize Via Posterior Samplingcited by 256
  9. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
  10. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  11. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 102
  12. 2015Contextual Dueling Banditscited by 101
33 more
  1. 2017Lower Bounds on Regret for Noisy Gaussian Process Bandit Optimizationcited by 78
  2. 2013A Literature Survey of Benchmark Functions For Global Optimization Problemscited by 77
  3. 2022Recent Advances in Bayesian Optimizationcited by 76
  4. 2017Preferential Bayesian Optimizationcited by 74
  5. 2014Reducing Dueling Bandits to Cardinal Banditscited by 62
  6. 2021Gaussian Process Bandit Optimization with Few Batchescited by 57
  7. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  8. 2021Optimal Order Simple Regret for Gaussian Process Banditscited by 54
  9. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  10. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  11. 2016Double Thompson Sampling for Dueling Banditscited by 42
  12. 2020Scalable Thompson Sampling using Sparse Gaussian Process Modelscited by 39
  13. 2020A Domain-Shrinking based Bayesian Optimization Algorithm with Order-Optimal Regret Performancecited by 39
  14. 2022Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Modelscited by 37
  15. 2015Copeland Dueling Banditscited by 36
  16. 2014Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper-Parameterscited by 31
  17. 2015Sparse Dueling Banditscited by 29
  18. 2024Feel-Good Thompson Sampling for Contextual Dueling Banditscited by 27
  19. 2023On the Sublinear Regret of GP-UCBcited by 26
  20. 2019Preference-Based Learning for Exoskeleton Gait Optimizationcited by 24
  21. 2023qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimizationcited by 23
  22. 2020Projective Preferential Bayesian Optimizationcited by 22
  23. 2020Preferential Batch Bayesian Optimizationcited by 19
  24. 2023Towards Practical Preferential Bayesian Optimization with Skew Gaussian Processescited by 19
  25. 2022Improved Convergence Rates for Sparse Approximation Methods in Kernel-Based Learningcited by 18
  26. 2023Sample Efficient Preference Alignment in LLMs via Active Explorationcited by 18
  27. 2024Principled Preferential Bayesian Optimizationcited by 16
  28. 2022Regret Bounds for Expected Improvement Algorithms in Gaussian Process Bandit Optimizationcited by 15
  29. 2019Zeroth Order Non-convex optimization with Dueling-Choice Banditscited by 11
  30. 2024Bandits with Preference Feedback: A Stackelberg Game Perspectivecited by 7
  31. 2013Exploiting correlation and budget constraints in Bayesian multi-armed bandit optimizationcited by 3
  32. 2025Bayesian Optimization from Human Feedback: Near-Optimal Regret Boundscited by 3
  33. 2025Preferential Multi-Objective Bayesian Optimization for Drug Discoverycited by 1

Built on by

Not cited yet.