mlsanityData to 14 Oct 2026

Dueling Posterior Sampling for Preference-Based Reinforcement Learning

arXiv 1908.01289 ↗First posted 4 Aug 2019cs.LG cs.AI stat.MLcited by 56

Not on the map. Why?

Cited by 33 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In preference-based reinforcement learning (RL), an agent interacts with the environment while receiving preferences instead of absolute feedback. While there is increasing research activity in preference-based RL, the design of formal frameworks that admit tractable theoretical analysis remains an open challenge. Building upon ideas from preference-based bandit learning and posterior sampling in RL, we present DUELING POSTERIOR SAMPLING (DPS), which employs preference-based posterior sampling to learn both the system dynamics and the underlying utility function that governs the preference feedback. As preference feedback is provided on trajectories rather than individual state-action pairs, we develop a Bayesian approach for the credit assignment problem, translating preferences to a posterior distribution over state-action reward models. We prove an asymptotic Bayesian no-regret rate for DPS with a Bayesian linear regression credit assignment model. This is the first regret guarantee for preference-based RL to our knowledge. We also discuss possible avenues for extending the proof methodology to other credit assignment models. Finally, we evaluate the approach empirically, showing competitive performance against existing baselines.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 2; 2023: 11; 2024: 16; 2025: 16; 2026: 7.

56 papers.

Sort:

By area:

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  4. 2023Nash Learning from Human Feedbackcited by 224
  5. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  6. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  7. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  8. 2023Is RLHF More Difficult than Standard RL?cited by 96
  9. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  10. 2021On the Expressivity of Markov Rewardcited by 71
  11. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  12. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
38 more
  1. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  2. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  3. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  4. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  5. 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
  6. 2024Feel-Good Thompson Sampling for Contextual Dueling Banditscited by 27
  7. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  8. 2024Optimal Design for Reward Modeling in RLHFcited by 25
  9. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 22
  10. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 21
  11. 2024Optimal Design for Human Preference Elicitationcited by 17
  12. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
  13. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problemscited by 13
  14. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  15. 2024Advances in Preference-based Reinforcement Learning: A Reviewcited by 8
  16. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  17. 2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedbackcited by 7
  18. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  19. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 6
  20. 2022Learning from Imperfect Demonstrations via Adversarial Confidence Transfercited by 5
  21. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 5
  22. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysiscited by 4
  23. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  24. 2024Preferential Multi-Objective Bayesian Optimizationcited by 3
  25. 2024On The Global Convergence Of Online RLHF With Neural Parametrizationcited by 3
  26. 2025Bayesian Optimization from Human Feedback: Near-Optimal Regret Boundscited by 3
  27. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3
  28. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  29. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  30. 2025Reinforcement Learning from Adversarial Preferences in Tabular MDPscited by 2
  31. 2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscited by 2
  32. 2026Learning Dynamics in RL Post-Training for Language Modelscited by 2
  33. 2026Online Learning and Equilibrium Computation with Ranking Feedbackcited by 2
  34. 2024DP-Dueling: Learning from Preference Feedback without Compromising User Privacycited by 1
  35. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hackingcited by 1
  36. 2026Offline Constrained RLHF with Multiple Preference Oraclescited by 1
  37. 2026Finding Stationary Points by Comparisonscited by 1
  38. 2023Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networkscited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
  2. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  3. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  4. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  5. 2026Minimax-Optimality of Posterior Sampling for Reinforcement Learning
  6. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  7. 2026UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning
  8. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?