mlsanityData to 7 Oct 2026

Best Policy Learning from Trajectory Preference Feedback

arXiv 2501.18873 ↗First posted 31 Jan 2025cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful approach for aligning generative models, but its reliance on learned reward models makes it vulnerable to mis-specification and reward hacking. Preference-based Reinforcement Learning (PbRL) offers a more robust alternative by directly leveraging noisy binary comparisons over trajectories. We study the best policy identification problem in PbRL, motivated by post-training optimization of generative models, for example, during multi-turn interactions. Learning in this setting combines an offline preference dataset - potentially biased or out-of-distribution and collected from a rater of subpar `competence' - with online pure exploration, making systematic online learning essential. To this end, we propose Posterior Sampling for Preference Learning (mathsfPSPL), a novel algorithm inspired by Top-Two Thompson Sampling that maintains posteriors over the reward model and dynamics. We provide the first Bayesian simple regret guarantees for PbRL and introduce an efficient approximation that outperforms existing baselines on simulation and image generation benchmarks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 86 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

50 of 86 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Optimal Design for Reward Modeling in RLHF22 Oct
  2. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning30 Jun
  3. 2024Online Bandit Learning with Offline Preference Data for Improved RLHF13 Jun
  4. 2024Multi-turn Reinforcement Learning from Preference Human Feedback23 May
  5. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards28 Feb
  6. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 40

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2016Concrete Problems in AI Safetycited by 1,582
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  8. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  9. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  12. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
28 more
  1. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  2. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  3. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  4. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  5. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  6. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  7. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  8. 2017Deep Exploration via Randomized Value Functionscited by 155
  9. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  10. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
  11. 2019Learning Adversarial MDPs with Bandit Feedback and Unknown Transitioncited by 84
  12. 2017Preferential Bayesian Optimizationcited by 72
  13. 2017Learning Unknown Markov Decision Processes: A Thompson Sampling Approachcited by 65
  14. 2017Ensemble Samplingcited by 62
  15. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  16. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  17. 2018Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in MDPscited by 41
  18. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?cited by 38
  19. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 35
  20. 2022Instance-Dependent Near-Optimal Policy Identification in Linear MDPs via Online Experiment Designcited by 33
  21. 2022Context-Aware Robust Fine-Tuningcited by 27
  22. 2019Preference-Based Learning for Exoskeleton Gait Optimizationcited by 23
  23. 2022An Analysis of Ensemble Samplingcited by 23
  24. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 20
  25. 2022Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrappingcited by 19
  26. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  27. 2023Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Datacited by 11
  28. 2023Bridging Imitation and Online Reinforcement Learning: An Optimistic Talecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  2. 2026Minimax-Optimality of Posterior Sampling for Reinforcement Learningcited by 0
  3. 2026Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  2. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  3. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation
  4. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  5. 2025Behavior Preference Regression for Offline Reinforcement Learning
  6. 2024In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning
  7. 2023Direct Preference-based Policy Optimization without Reward Modeling
  8. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learning