mlsanityData to 7 Oct 2026

Dataset Reset Policy Optimization for RLHF

arXiv 2404.08495 ↗First posted 12 Apr 2024cs.LG cs.AI cs.CLcited by 41

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning (RL) from Human Preference-based feedback is a popular paradigm for fine-tuning generative models, which has produced impressive models such as GPT-4 and Claude3 Opus. This framework often consists of two steps: learning a reward model from an offline preference dataset followed by running online RL to optimize the learned reward model. In this work, leveraging the idea of reset, we propose a new RLHF algorithm with provable guarantees. Motivated by the fact that offline preference dataset provides informative states (i.e., data that is preferred by the labelers), our new algorithm, Dataset Reset Policy Optimization (DR-PO), integrates the existing offline preference dataset into the online policy training procedure via dataset reset: it directly resets the policy optimizer to the states in the offline dataset, instead of always starting from the initial state distribution. In theory, we show that DR-PO learns to perform at least as good as any policy that is covered by the offline dataset under general function approximation with finite sample complexity. In experiments, we demonstrate that on both the TL;DR summarization and the Anthropic Helpful Harmful (HH) dataset, the generation from DR-PO is better than that from Proximal Policy Optimization (PPO) and Direction Preference Optimization (DPO), under the metric of GPT4 win-rate. Code for this work can be found at https://github.com/Cornell-RL/drpo.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 65 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

47 of 65 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Self-Rewarding Language Models18 Jan
  2. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  3. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment30 Sep
  4. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
  5. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  8. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  10. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  11. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  11. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  12. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
23 more
  1. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  3. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  4. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  5. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  6. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  7. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  8. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  9. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  10. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  11. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  12. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  13. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  14. 2022Jump-Start Reinforcement Learningcited by 102
  15. 2015Contextual Dueling Banditscited by 101
  16. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  17. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  18. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  19. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  20. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  21. 2023Learning to Generate Better Than Your LLMcited by 49
  22. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  23. 2021Efficient Local Planning with Linear Function Approximationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 14; 2026: 11.

41 papers.

Sort:

By area:

  1. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  2. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  3. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  4. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  5. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  6. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  7. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  8. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  9. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  10. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  11. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 57
  12. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
29 more
  1. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  2. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 39
  3. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  4. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  5. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  6. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
  7. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  8. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumcited by 22
  9. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 18
  10. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  11. 2024Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Samplingcited by 13
  12. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexitycited by 11
  13. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  14. 2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 9
  15. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  16. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  17. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 6
  18. 2026Towards a Theoretical Understanding to the Generalization of RLHFcited by 5
  19. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  20. 2025Efficient Imitation under Misspecificationcited by 4
  21. 2024Annotation-Efficient Language Model Alignment via Diverse and Representative Response Textscited by 3
  22. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
  23. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  24. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  25. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1
  26. 2026Response Time Enhances Alignment with Heterogeneous Preferencescited by 1
  27. 2026Credit Assignment with Resets in Language Model Reasoningcited by 1
  28. 2026Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Explorationcited by 0
  29. 2026Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  2. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  3. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  4. 2025Design Considerations in Offline Preference-based RL
  5. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  6. 2024Online Bandit Learning with Offline Preference Data for Improved RLHF
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2021A Minimalist Approach to Offline Reinforcement Learning