Data to 7 Oct 2026

Is RLHF More Difficult than Standard RL?

arXiv 2306.14111 ↗First posted 25 Jun 2023cs.LG cs.AI stat.MLcited by 96

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from Human Feedback (RLHF) learns from preference signals, while standard Reinforcement Learning (RL) directly learns from reward signals. Preferences arguably contain less information than rewards, which makes preference-based RL seemingly more difficult. This paper theoretically proves that, for a wide range of preference models, we can solve preference-based RL directly using existing algorithms and techniques for reward-based RL, with small or no extra costs. Specifically, (1) for preferences that are drawn from reward-based probabilistic models, we reduce the problem to robust reward-based RL that can tolerate small errors in rewards; (2) for general arbitrary preferences where the objective is to find the von Neumann winner, we reduce the problem to multiagent reward-based RL which finds Nash equilibria for factored Markov games with a restricted set of policies. The latter case can be further reduced to adversarial MDP when preferences only depend on the final state. We instantiate all reward-based RL subroutines by concrete provable algorithms, and apply our theory to a large class of models including tabular MDPs and MDPs with generic function approximation. We further provide guarantees when K-wise comparisons are available.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 56 references that are in the map. First posted on arXiv 25 Jun 2023.

Builds on

32 of 56 references.

Closest ancestors · 7

Same direction, newest first

Other references · 25

2022Training language models to follow instructions with human feedbackcited by 15,712
2013Playing Atari with Deep Reinforcement Learningcited by 3,211
2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
2017Minimax Regret Bounds for Reinforcement Learningcited by 500
2023Aligning Text-to-Image Models using Human Feedbackcited by 418
2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
2021The Statistical Complexity of Interactive Decision Makingcited by 208
2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
2020Reward-Free Exploration for Reinforcement Learningcited by 177
2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122

13 more

2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
2015Contextual Dueling Banditscited by 101
2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
2014Reducing Dueling Bandits to Cardinal Banditscited by 62
2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
2018PAC Battling Bandits in the Plackett-Luce Modelcited by 27
2022Sample-Efficient Reinforcement Learning of Partially Observable Markov Gamescited by 26
2023Improved Regret for Efficient Online Reinforcement Learning with Linear Function Approximationcited by 16
2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10
2015Consistent Probabilistic Social Choicecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 40; 2025: 32; 2026: 18.

96 papers.

Sort:

By area:

2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
2023Nash Learning from Human Feedbackcited by 220
2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78

38 more

2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
2024Reinforcement Learning from Human Feedback with Active Queriescited by 39
2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24
2024Optimal Design for Reward Modeling in RLHFcited by 23
2024PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedbackcited by 22
2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
2024Optimal Design for Human Preference Elicitationcited by 17
2024Principled Preferential Bayesian Optimizationcited by 16
2024Understanding the Learning Dynamics of Alignment with Human Feedbackcited by 15
2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
2023Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedbackcited by 13
2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
2024Fakes of Varying Shades: How Warning Affects Human Perception and Engagement Regarding LLM Hallucinationscited by 12
2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12
2025Unsupervised Feature Transformation via In-context Generation, Generator-critic LLM Agents, and Duet-play Teamingcited by 11
2024Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedbackcited by 10
2025Greedy Sampling Is Provably Efficient for RLHFcited by 10
2025Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospectscited by 9
2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
2023Demonstration-Regularized RLcited by 8
2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedbackcited by 7
2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
2025Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologiescited by 5

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way