mlsanityData to 7 Oct 2026

Policy-labeled Preference Learning: Is Preference Enough for RLHF?

arXiv 2505.06273 ↗First posted 6 May 2025cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

To design rewards that align with human goals, Reinforcement Learning from Human Feedback (RLHF) has emerged as a prominent technique for learning reward functions from human preferences and optimizing policies via reinforcement learning algorithms. However, existing RLHF methods often misinterpret trajectories as being generated by an optimal policy, causing inaccurate likelihood estimation and suboptimal learning. Inspired by Direct Preference Optimization framework which directly learns optimal policy without explicit reward, we propose policy-labeled preference learning (PPL), to resolve likelihood mismatch issues by modeling human preferences with regret, which reflects behavior policy information. We also provide a contrastive KL regularization, derived from regret-based principles, to enhance RLHF in sequential decision making. Experiments in high-dimensional continuous control tasks demonstrate PPL's significant improvements in offline RLHF performance and its effectiveness in online settings.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 41 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

34 of 41 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models23 Oct
  2. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  3. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF29 Apr
  4. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences4 Mar
  5. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  7. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  8. 2023Learning Optimal Advantage from Preferences and Mistaking it for Reward3 Oct
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2023Inverse Preference Learning: Preference-based RL without a Reward Function24 May
  11. 2023Direct Preference-based Policy Optimization without Reward Modeling30 Jan
  12. 2022Models of human preference for learning reward functions5 Jun
4 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  3. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
6 more
  1. 2024Token-level Direct Preference Optimizationcited by 135
  2. 2020Quantifying Differences in Reward Functionscited by 62
  3. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  4. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  5. 2024Aligning Language Models with Demonstrated Feedbackcited by 32
  6. 2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Regret Minimization Framework on Preference Learning in Large Language Modelscited by 0
  2. 2026PAWS: Preference Learning with Advantage-Weighted Segmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Is RLHF More Difficult than Standard RL?
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling
  4. 2025Influencing Humans to Conform to Preference Models for RLHF
  5. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  6. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
  7. 2024Dataset Reset Policy Optimization for RLHF
  8. 2025Greedy Sampling Is Provably Efficient for RLHF