mlsanityData to 7 Oct 2026

Online Iterative Reinforcement Learning from Human Feedback with General Preference Model

arXiv 2402.07314 ↗First posted 11 Feb 2024cs.LG stat.MLcited by 52

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We investigate Reinforcement Learning from Human Feedback (RLHF) in the context of a general preference oracle. In particular, we do not assume the existence of a reward function and an oracle preference signal drawn from the Bradley-Terry model as most of the prior works do. We consider a standard mathematical formulation, the reverse-KL regularized minimax game between two LLMs for RLHF under general preference oracle. The learning objective of this formulation is to find a policy so that it is consistently preferred by the KL-regularized preference oracle over any competing LLMs. We show that this framework is strictly more general than the reward-based one, and propose sample-efficient algorithms for both the offline learning from a pre-collected preference dataset and online learning where we can query the preference oracle along the way of training. Empirical studies verify the effectiveness of the proposed framework.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 59 of its 91 references that are in the map. First posted on arXiv 11 Feb 2024.

Builds on

59 of 91 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024Self-Rewarding Language Models18 Jan
  2. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  4. 2023Nash Learning from Human Feedback1 Dec
  5. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  6. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  7. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  8. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  9. 2023Is RLHF More Difficult than Standard RL?25 Jun
  10. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
8 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  4. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  11. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  12. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
27 more
  1. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  2. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  3. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  6. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  7. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  8. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  9. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  10. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  11. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  12. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  13. 2015Contextual Dueling Banditscited by 101
  14. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
  15. 2022The Role of Coverage in Online Reinforcement Learningcited by 82
  16. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  17. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  18. 2021Feel-Good Thompson Sampling for Contextual Bandits and Reinforcement Learningcited by 65
  19. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  20. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 53
  21. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 45
  22. 2022VOQL: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximationcited by 44
  23. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  24. 2022Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processescited by 33
  25. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  26. 2023Offline Learning in Markov Games with General Function Approximationcited by 16
  27. 2022Fast Rates in Pool-Based Batch Active Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 30; 2026: 19.

52 papers.

Sort:

By area:

  1. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  2. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  3. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  4. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  5. 2025Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reusecited by 28
  6. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  7. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspectivecited by 16
  8. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  9. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  10. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  11. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  12. 2025Reward Model Overoptimisation in Iterated RLHFcited by 10
38 more
  1. 2025Game Theory Meets LLM and Agentic AI: Reimagining Cybersecurity for the Age of Intelligent Threatscited by 10
  2. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 10
  3. 2024Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learningcited by 7
  4. 2025Mitigating Preference Hacking in Policy Optimization with Pessimismcited by 7
  5. 2025Multiplayer Nash Preference Optimizationcited by 7
  6. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 6
  7. 2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
  8. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  9. 2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Samplingcited by 4
  10. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  11. 2025Evolutionary Guided Decoding: Iterative Value Refinement for LLMscited by 3
  12. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 3
  13. 2025Offline and Online KL-Regularized RLHF under Differential Privacycited by 3
  14. 2026Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modelingcited by 3
  15. 2026Pessimism-Free Offline Learning in General-Sum Games via KL Regularizationcited by 3
  16. 2026Offline Two-Player Zero-Sum Markov Games with KL Regularizationcited by 3
  17. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?cited by 2
  18. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
  19. 2025PITA: Preference-Guided Inference-Time Alignment for LLM Post-Trainingcited by 2
  20. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analysescited by 2
  21. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  22. 2026Autoregressive Direct Preference Optimizationcited by 1
  23. 2026Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecificationcited by 1
  24. 2024iREPO: implicit Reward Pairwise Difference based Empirical Preference Optimizationcited by 0
  25. 2025Online Knowledge Distillation with Reward Guidancecited by 0
  26. 2025Offline Clustering of Preference Learning with Active-data Augmentationcited by 0
  27. 2025Bootstrapping LLMs via Preference-Based Policy Optimizationcited by 0
  28. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0
  29. 2025Unregularized Linear Convergence in Zero-Sum Game from Preference Feedbackcited by 0
  30. 2026Demystifying the unreasonable effectiveness of online alignment methodscited by 0
  31. 2026IRIS: Interpolative R'enyi Iterative Self-play for Large Language Model Fine-Tuningcited by 0
  32. 2026Personalized Alignment Revisited: The Necessity and Sufficiency of User Diversitycited by 0
  33. 2026Truthful Online Preference Aggregation for LLM Fine-Tuning in Mobile Crowdsourcingcited by 0
  34. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 0
  35. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  36. 2026LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedbackcited by 0
  37. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  38. 2026When Greedy Sampling Explores: KL-Regularized Contextual Bandits without Eluder-Dimension Dependencecited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Offline Constrained RLHF with Multiple Preference Oracles
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  3. 2024Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  5. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  6. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2025Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference