mlsanityData to 14 Oct 2026

Provable Reward-Agnostic Preference-Based Reinforcement Learning

arXiv 2305.18505 ↗First posted 29 May 2023cs.LG cs.AI math.STcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories, rather than explicit reward signals. While PbRL has demonstrated practical success in fine-tuning language models, existing theoretical work focuses on regret minimization and fails to capture most of the practical frameworks. In this study, we fill in such a gap between theoretical PbRL and practical algorithms by proposing a theoretical reward-agnostic PbRL framework where exploratory trajectories that enable accurate learning of hidden reward functions are acquired before collecting any human feedback. Theoretical analysis demonstrates that our algorithm requires less human feedback for learning the optimal policy under preference-based models with linear parameterization and unknown transitions, compared to the existing theoretical literature. Specifically, our framework can incorporate linear and low-rank MDPs with efficient sample complexity. Additionally, we investigate reward-agnostic RL with action-based comparison feedback and introduce an efficient querying algorithm tailored to this scenario.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 47 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

35 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  2. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  3. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  5. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  6. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  8. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  9. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  10. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  11. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  12. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
17 more
  1. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  2. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
  3. 2020Effective Diversity in Population Based Reinforcement Learningcited by 119
  4. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  5. 2020Invariant Causal Prediction for Block MDPscited by 105
  6. 2015Contextual Dueling Banditscited by 101
  7. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  8. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  9. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  10. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  11. 2020Provably Efficient Reward-Agnostic Navigation with Linear Value Iterationcited by 56
  12. 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
  13. 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
  14. 2022On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RLcited by 27
  15. 2021Block Contextual MDPs for Continual Learningcited by 25
  16. 2021On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Gamecited by 20
  17. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 10; 2026: 5.

25 papers.

Sort:

By area:

  1. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  3. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  4. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  5. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  6. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  7. 2024Optimal Design for Human Preference Elicitationcited by 17
  8. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  9. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 11
  10. 2023Demonstration-Regularized RLcited by 9
  11. 2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedbackcited by 7
  12. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
13 more
  1. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 6
  2. 2025Improved Bounds for Private and Robust Alignmentcited by 5
  3. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
  4. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3
  5. 2023Contextual Bandits and Imitation Learning via Preference-Based Active Queriescited by 2
  6. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  7. 2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscited by 2
  8. 2026Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedbackcited by 2
  9. 2024A Theoretical Framework for Partially Observed Reward-States in RLHFcited by 1
  10. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 1
  11. 2025Reinforcement Learning from Multi-level and Episodic Human Feedbackcited by 0
  12. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  13. 2026UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Direct Preference-based Policy Optimization without Reward Modeling
  2. 2024Advances in Preference-based Reinforcement Learning: A Review
  3. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  4. 2024RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning
  5. 2023Data Driven Reward Initialization for Preference based Reinforcement Learning
  6. 2026SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
  7. 2024Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model
  8. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learning