mlsanityData to 7 Oct 2026

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

arXiv 2312.11456 ↗First posted 18 Dec 2023cs.LG cs.AI stat.MLcited by 346

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

This paper studies the alignment process of generative models with Reinforcement Learning from Human Feedback (RLHF). We first identify the primary challenges of existing popular methods like offline PPO and offline DPO as lacking in strategical exploration of the environment. Then, to understand the mathematical principle of RLHF, we consider a standard mathematical formulation, the reverse-KL regularized contextual bandit for RLHF. Despite its widespread practical application, a rigorous theoretical analysis of this formulation remains open. We investigate its behavior in three distinct settings -- offline, online, and hybrid -- and propose efficient algorithms with finite-sample theoretical guarantees. Moving towards practical applications, our framework, with a robust approximation of the information-theoretical policy improvement oracle, naturally gives rise to several novel RLHF algorithms. This includes an iterative version of the Direct Preference Optimization (DPO) algorithm for online settings, and a multi-step rejection sampling strategy for offline scenarios. Our empirical evaluations on real-world alignment experiment of large language model demonstrate that these proposed methods significantly surpass existing strong baselines, such as DPO and Rejection Sampling Optimization (RSO), showcasing the connections between solid theoretical foundations and their potent practical implementations.

Key facts. Cited by 346 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 76 of its 104 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

76 of 104 references.

Closest ancestors · 20

Same direction, newest first

  1. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  2. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  3. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  4. 2023Enabling Language Models to Implicitly Learn Self-Improvement2 Oct
  5. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  7. 2023Is RLHF More Difficult than Standard RL?25 Jun
  8. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
8 more
  1. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  2. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  3. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  4. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 56

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2021Convex optimizationcited by 2,479
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  10. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  11. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  12. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
38 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  2. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  3. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  6. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  7. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  8. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  9. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  10. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  11. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  12. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  13. 2020Improved Optimistic Algorithms for Logistic Banditscited by 117
  14. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  15. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  16. 2020Batch Value-function Approximation with Only Realizabilitycited by 103
  17. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
  18. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  19. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  20. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  21. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  22. 2021Feel-Good Thompson Sampling for Contextual Bandits and Reinforcement Learningcited by 65
  23. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
  24. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
  25. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  26. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  27. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 53
  28. 2021The Power of Exploiter: Provable Multi-Agent RL in Large State Spacescited by 48
  29. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  30. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 45
  31. 2022VOQL: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximationcited by 44
  32. 2021Towards General Function Approximation in Zero-Sum Markov Gamescited by 42
  33. 2020Understanding Learned Reward Functionscited by 37
  34. 2022Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processescited by 33
  35. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  36. 2022Nearly Minimax Optimal Reinforcement Learning with Linear Function Approximationcited by 30
  37. 2023A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processescited by 28
  38. 2020Optimistic Exploration even with a Pessimistic Initialisationcited by 25

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 108; 2025: 153; 2026: 85.

346 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  3. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  5. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  6. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  7. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  8. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  9. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  10. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  12. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
38 more
  1. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  4. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  5. 2024Learning Dynamics of LLM Finetuningcited by 93
  6. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  7. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  8. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  9. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  10. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  11. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  12. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  13. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  14. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  15. 2025Self-rewarding correction for mathematical reasoningcited by 67
  16. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  17. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  18. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  19. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  20. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  21. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  22. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  23. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  24. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  25. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  26. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  27. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  28. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  29. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  30. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  31. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  32. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  33. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  34. 2025DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Drivingcited by 39
  35. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  36. 2024Entropy-Regularized Process Reward Modelcited by 38
  37. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  38. 2025A Survey of Direct Preference Optimizationcited by 35

50 most cited of 346 All 346 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  3. 2024Active Preference Optimization for Sample Efficient RLHF
  4. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling
  5. 2023A Survey of Reinforcement Learning from Human Feedback
  6. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  7. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
  8. 2024Optimal Design for Reward Modeling in RLHF