mlsanityData to 14 Oct 2026

DPO Meets PPO: Reinforced Token Optimization for RLHF

arXiv 2404.18922 ↗First posted 29 Apr 2024cs.LG cs.AI cs.CLcited by 127

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In the classical Reinforcement Learning from Human Feedback (RLHF) framework, Proximal Policy Optimization (PPO) is employed to learn from sparse, sentence-level rewards -- a challenging scenario in traditional deep reinforcement learning. Despite the great successes of PPO in the alignment of large language models, its open-source implementation is still largely sub-optimal. To address these issues, we introduce a framework that models RLHF problems as a Markov decision process (MDP), enabling the capture of fine-grained token-wise information. Under this framework, we introduce an algorithm Reinforced Token Optimization (textttRTO), which learns the token-wise reward function from preference data and performs policy optimization based on this learned token-wise reward signal. Theoretically, textttRTO is proven to have the capability of finding the near-optimal policy sample-efficiently. For its practical implementation, textttRTO innovatively integrates Direct Preference Optimization (DPO) and PPO. DPO, originally derived from sparse sentence rewards, surprisingly provides us with a token-wise characterization of response quality, which is seamlessly incorporated into our subsequent PPO training stage. Extensive experiments demonstrate that textttRTO performs better than PPO and other direct preference learning algorithms. In particular, RTO outperforms PPO by 7.5 points on the AlpacaEval 2 benchmark and by 4.1 points on Arena-Hard. Our code and models are available at hrefhttps://github.com/zkshan2002/RTOhttps://github.com/zkshan2002/RTO.

Key facts. Cited by 127 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 82 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

56 of 82 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  3. 2024Dense Reward for Free in Reinforcement Learning from Human Feedback1 Feb
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  5. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  9. 2023Is RLHF More Difficult than Standard RL?25 Jun
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  11. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
7 more
  1. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2017Hindsight Experience Replaycited by 1,136
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  12. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
25 more
  1. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  4. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  5. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  6. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
  7. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  8. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  9. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  10. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  11. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  12. 2024Token-level Direct Preference Optimizationcited by 135
  13. 2020Improved Optimistic Algorithms for Logistic Banditscited by 118
  14. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  15. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 102
  16. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  17. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  18. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  19. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 55
  20. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 46
  21. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  22. 2023A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processescited by 29
  23. 2023Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Explorationcited by 20
  24. 2023On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximationcited by 14
  25. 2021Nearly Optimal Policy Optimization with Stable at Any Time Guaranteecited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 35; 2025: 63; 2026: 29.

127 papers.

Sort:

By area:

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  3. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  4. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  5. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  6. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  7. 2024Process Reward Model with Q-Value Rankingscited by 80
  8. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  9. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  10. 2025Self-rewarding correction for mathematical reasoningcited by 70
  11. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  12. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
38 more
  1. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuningcited by 59
  2. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  3. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  4. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 51
  5. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 50
  6. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 47
  7. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 44
  8. 2024Entropy-Regularized Process Reward Modelcited by 38
  9. 2025A Survey of Direct Preference Optimizationcited by 36
  10. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 34
  11. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
  12. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  13. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  14. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
  15. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Modelscited by 28
  16. 2025StepWiser: Stepwise Generative Judges for Wiser Reasoningcited by 28
  17. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 27
  18. 2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
  19. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  20. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 24
  21. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  22. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 22
  23. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspectivecited by 17
  24. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillationcited by 17
  25. 2025Discriminative Policy Optimization for Token-Level Reward Modelscited by 16
  26. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Modelscited by 15
  27. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  28. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  29. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 14
  30. 2025Token-Importance Guided Direct Preference Optimizationcited by 14
  31. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Frameworkcited by 14
  32. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
  33. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12
  34. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  35. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Trainingcited by 12
  36. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learningcited by 11
  37. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 11
  38. 2026Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learningcited by 11

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
  2. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlap
  3. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  4. 2023Proximal Policy Optimization Actual Combat: Manipulating Output Tokenizer Length
  5. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
  6. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  7. 2024Filtered Direct Preference Optimization
  8. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment