mlsanityData to 7 Oct 2026

Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization

arXiv 2402.10342 ↗First posted 15 Feb 2024cs.LG cs.AI cs.CLcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has achieved impressive empirical successes while relying on a small amount of human feedback. However, there is limited theoretical justification for this phenomenon. Additionally, most recent studies focus on value-based algorithms despite the recent empirical successes of policy-based algorithms. In this work, we consider an RLHF algorithm based on policy optimization (PO-RLHF). The algorithm is based on the popular Policy Cover-Policy Gradient (PC-PG) algorithm, which assumes knowledge of the reward function. In PO-RLHF, knowledge of the reward function is not assumed, and the algorithm uses trajectory-based comparison feedback to infer the reward function. We provide performance bounds for PO-RLHF with low query complexity, which provides insight into why a small amount of human feedback may be sufficient to achieve good performance with RLHF. A key novelty is a trajectory-level elliptical potential analysis, which bounds the reward estimation error when comparison feedback (rather than numerical reward observation) is given. We provide and analyze algorithms PG-RLHF and NN-PG-RLHF for two settings: linear and neural function approximation, respectively.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 37 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

19 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  3. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  4. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems24 Jul
  5. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
  6. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  7. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 11

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  5. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  6. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  7. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  8. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
  9. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  10. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  11. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 9.

25 papers.

Sort:

By area:

  1. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  2. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  3. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  4. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  5. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  6. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  7. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  8. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  9. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
  10. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysiscited by 4
  11. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  12. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
13 more
  1. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  2. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  3. 2024DOPL: Direct Online Preference Learning for Restless Bandits with Preference Feedbackcited by 2
  4. 2025Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generationcited by 2
  5. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  6. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholdscited by 2
  7. 2026Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedbackcited by 2
  8. 2026Online Learning and Equilibrium Computation with Ranking Feedbackcited by 1
  9. 2026Regret Bounds for Reinforcement Learning from Multi-Source Imperfect Preferencescited by 1
  10. 2026Offline Constrained RLHF with Multiple Preference Oraclescited by 0
  11. 2026Distributed Zeroth-Order Policy Gradient for Networked Multi-agent Reinforcement Learning from Human Feedbackcited by 0
  12. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  13. 2026Online Reward-Punishment Learning from Fixed-Channel Perceptual Event Streams without Environment Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning from Human Feedback
  2. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2025One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF
  5. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  6. 2026Towards a Theoretical Understanding to the Generalization of RLHF
  7. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  8. 2023Contrastive Preference Learning: Learning from Human Feedback without RL