mlsanityData to 7 Oct 2026

Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems

arXiv 2307.12975 ↗First posted 24 Jul 2023cs.LG math.ST stat.MLcited by 12

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

For a real-world decision-making problem, the reward function often needs to be engineered or learned. A popular approach is to utilize human feedback to learn a reward function for training. The most straightforward way to do so is to ask humans to provide ratings for state-action pairs on an absolute scale and take these ratings as reward samples directly. Another popular way is to ask humans to rank a small set of state-action pairs by preference and learn a reward function from these preference data. Recently, preference-based methods have demonstrated substantial success in empirical applications such as InstructGPT. In this work, we develop a theoretical comparison between these human feedback approaches in offline contextual bandits and show how human bias and uncertainty in feedback modelings can affect the theoretical guarantees of these approaches. Through this, our results seek to provide a theoretical explanation for the empirical successes of preference-based methods from a modeling perspective.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 74 references that are in the map. First posted on arXiv 24 Jul 2023.

Builds on

36 of 74 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  2. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  5. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  8. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  9. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  10. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  11. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 82
  12. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
21 more
  1. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  2. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 69
  3. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 67
  4. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  5. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
  6. 2014Reducing Dueling Bandits to Cardinal Banditscited by 62
  7. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 59
  8. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  9. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  10. 2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 51
  11. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  12. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  13. 2016A Relative Exponential Weighing Algorithm for Adversarial Utility-based Dueling Banditscited by 31
  14. 2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
  15. 2020A Simple Model for Subject Behavior in Subjective Experimentscited by 14
  16. 2021Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settingscited by 14
  17. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  18. 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
  19. 2017Predicting Aesthetic Score Distribution through Cumulative Jensen-Shannon Divergencecited by 9
  20. 2016Recover Subjective Quality Scores from Noisy Measurementscited by 8
  21. 2020GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labelingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 4; 2026: 1.

12 papers.

  1. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  2. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  3. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  4. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  5. 2024Feel-Good Thompson Sampling for Contextual Dueling Banditscited by 23
  6. 2024Bandits with Preference Feedback: A Stackelberg Game Perspectivecited by 7
  7. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6
  8. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  9. 2024Conversational Dueling Bandits in Generalized Linear Modelscited by 5
  10. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  11. 2025Fusing Reward and Dueling Feedback in Stochastic Banditscited by 1
  12. 2026On the Complexity of Preference-Based Banditscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Neural Dueling Bandits: Preference-Based Optimization with Human Feedback
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  3. 2023Kernelized Offline Contextual Dueling Bandits
  4. 2022Models of human preference for learning reward functions
  5. 2025Capturing Individual Human Preferences with Reward Features
  6. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning