Data to 7 Oct 2026

Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons

arXiv 2301.11270 ↗First posted 26 Jan 2023cs.LG cs.AI cs.HCcited by 244

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We provide a theoretical framework for Reinforcement Learning with Human Feedback (RLHF). Our analysis shows that when the true reward function is linear, the widely used maximum likelihood estimator (MLE) converges under both the Bradley-Terry-Luce (BTL) model and the Plackett-Luce (PL) model. However, we show that when training a policy based on the learned reward model, MLE fails while a pessimistic MLE provides policies with improved performance under certain coverage assumptions. Additionally, we demonstrate that under the PL model, the true MLE and an alternative MLE that splits the K-wise comparison into pairwise comparisons both converge. Moreover, the true MLE is asymptotically more efficient. Our results validate the empirical success of existing RLHF algorithms in InstructGPT and provide new insights for algorithm design. Furthermore, our results unify the problem of RLHF and max-entropy Inverse Reinforcement Learning (IRL), and provide the first sample complexity bound for max-entropy IRL.

Key facts. Cited by 244 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 103 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

53 of 103 references.

Closest ancestors · 6

Same direction, newest first

Other references · 47

2022Training language models to follow instructions with human feedbackcited by 15,712
2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
2020Learning to summarize from human feedbackcited by 2,562
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2016Generative Adversarial Imitation Learningcited by 1,576
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
2022Improving alignment of dialogue agents via targeted human judgementscited by 527
2021Implicit Behavioral Cloningcited by 442
2020Is Pessimism Provably Efficient for Offline RL?cited by 362
2022Teaching language models to support answers with verified quotescited by 290
2021Recursively Summarizing Books with Human Feedbackcited by 287

35 more

2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
2020Improved Optimistic Algorithms for Logistic Banditscited by 117
2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
2014Best-Arm Identification in Linear Banditscited by 98
2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
2015Simple, Robust and Optimal Ranking from Pairwise Comparisonscited by 68
2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
2014Reducing Dueling Bandits to Cardinal Banditscited by 62
2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 59
2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 51
2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
2015Spectral MLE: Top-K Rank Aggregation from Pairwise Comparisonscited by 34
2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
2016A Relative Exponential Weighing Algorithm for Adversarial Utility-based Dueling Banditscited by 31
2018PAC Battling Bandits in the Plackett-Luce Modelcited by 27
2014Minimax-optimal Inference from Partial Rankingscited by 22
2018Approximate Ranking from Pairwise Comparisonscited by 21
2018Active Ranking with Subset-wise Preferencescited by 21
2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
2022When is Realizability Sufficient for Off-Policy Reinforcement Learning?cited by 17
2017Minimax Rates and Efficient Algorithms for Noisy Sortingcited by 15
2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
2017Learning from Comparisons and Choicescited by 12
2022Exploiting Correlation to Achieve Faster Learning Rates in Low-Rank Preference Banditscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 43; 2024: 83; 2025: 75; 2026: 43.

244 papers.

Sort:

By area:

2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2023Preference Ranking Optimization for Human Alignmentcited by 283
2023Large Language Model Alignment: A Surveycited by 241
2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
2023Mitigating the Alignment Tax of RLHFcited by 181
2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133

38 more

2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
2024Critique-out-Loud Reward Modelscited by 105
2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
2023Is RLHF More Difficult than Standard RL?cited by 96
2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
2025Outcome-based Exploration for LLM Reasoningcited by 79
2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
2024Robust Preference Optimization through Reward Model Distillationcited by 67
2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
2024Active Preference Optimization for Sample Efficient RLHFcited by 61
2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
2023Provable Offline Preference-Based Reinforcement Learningcited by 55
2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
2024Axioms for AI Alignment from Human Feedbackcited by 50
2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
2024Beyond Preferences in AI Alignmentcited by 46
2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
2024Dataset Reset Policy Optimization for RLHFcited by 41
2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40
2024Reinforcement Learning from Human Feedback with Active Queriescited by 39
2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
2023The Trickle-down Impact of Reward (In-)consistency on RLHFcited by 29
2023Human-Aligned Calibration for AI-Assisted Decision Makingcited by 28

50 most cited of 244 All 244 (JSON)

Similar, not cited

Close in wording, no citation either way