mlsanityData to 7 Oct 2026

Direct Preference-based Policy Optimization without Reward Modeling

arXiv 2301.12842 ↗First posted 30 Jan 2023cs.LG cs.AIcited by 36

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Preference-based reinforcement learning (PbRL) is an approach that enables RL agents to learn from preference, which is particularly useful when formulating a reward function is challenging. Existing PbRL methods generally involve a two-step procedure: they first learn a reward model based on given preference data and then employ off-the-shelf reinforcement learning algorithms using the learned reward model. However, obtaining an accurate reward model solely from preference information, especially when the preference is from human teachers, can be difficult. Instead, we propose a PbRL algorithm that directly learns from preference without requiring any reward modeling. To achieve this, we adopt a contrastive learning framework to design a novel policy scoring metric that assigns a high score to policies that align with the given preferences. We apply our algorithm to offline RL tasks with actual human preference labels and show that our algorithm outperforms or is on par with the existing PbRL methods. Notably, on high-dimensional control tasks, our algorithm surpasses offline RL methods that learn with ground-truth reward information. Finally, we show that our algorithm can be successfully applied to fine-tune large language models.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 60 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

43 of 60 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning30 May
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  3. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  4. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  6. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2020Supervised Contrastive Learningcited by 2,803
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
24 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2020Graph Contrastive Learning with Augmentationscited by 1,203
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2019When Does Label Smoothing Help?cited by 985
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2022A Generalist Agentcited by 802
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  13. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  14. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  15. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  16. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  17. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  18. 2022Multi-Game Decision Transformerscited by 212
  19. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  20. 2021Offline RL Without Off-Policy Evaluationcited by 150
  21. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  22. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  23. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  24. 2021Offline Preference-Based Apprenticeship Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 13; 2026: 8.

36 papers.

Sort:

By area:

  1. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  3. 2024Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generationcited by 64
  4. 2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 28
  5. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  6. 2024Improving Attributed Text Generation of Large Language Models via Preference Learningcited by 23
  7. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  8. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  9. 2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
  10. 2024Preference Aligned Diffusion Planner for Quadrupedal Locomotion Controlcited by 7
  11. 2024Aligning Transformers with Continuous Feedback via Energy Rank Alignmentcited by 6
  12. 2024Boltzmann-Aligned Inverse Folding Model as a Predictor of Mutational Effects on Protein-Protein Interactionscited by 5
24 more
  1. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferencescited by 4
  2. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  3. 2023Conditions on Preference Relations that Guarantee the Existence of Optimal Policiescited by 3
  4. 2024Hindsight Preference Learning for Offline Preference-based Reinforcement Learningcited by 3
  5. 2025Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Modelscited by 3
  6. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  7. 2025From Action Labels to Sets: Rethinking Action Supervision for Imitation Learning from Corrective Feedbackcited by 2
  8. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  9. 2025Policy Teaching via Data Poisoning in Learning from Human Preferencescited by 2
  10. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?cited by 2
  11. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
  12. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learningcited by 2
  13. 2026Causally Robust Reward Learning from Reason-Augmented Preference Feedbackcited by 2
  14. 2024Evaluation of Few-Shot Learning for Classification Tasks in the Polish Languagecited by 1
  15. 2024Chain of Alignment: Integrating Public Will with Expert Intelligence for Language Model Alignmentcited by 1
  16. 2025Behavior Preference Regression for Offline Reinforcement Learningcited by 1
  17. 2026RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedbackcited by 1
  18. 2026P2-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimizationcited by 1
  19. 2025TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learningcited by 0
  20. 2026OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Explorationcited by 0
  21. 2026Implicit Safety Alignment from Crowd Preferencescited by 0
  22. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 0
  23. 2026A Regret Minimization Framework on Preference Learning in Large Language Modelscited by 0
  24. 2026PAWS: Preference Learning with Advantage-Weighted Segmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  2. 2025Offline Safe Policy Optimization From Heterogeneous Feedback
  3. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  4. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  5. 2026SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
  6. 2025Diffusion Classifier-Driven Reward for Offline Preference-based Reinforcement Learning
  7. 2023Beyond Reward: Offline Preference-guided Policy Optimization
  8. 2021Dueling RL: Reinforcement Learning with Trajectory Preferences