mlsanityData to 7 Oct 2026

Provable Offline Preference-Based Reinforcement Learning

arXiv 2305.14816 ↗First posted 24 May 2023cs.LG math.ST stat.MLcited by 55

Not on the map. Why?

Cited by 42 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we investigate the problem of offline Preference-based Reinforcement Learning (PbRL) with human feedback where feedback is available in the form of preference between trajectory pairs rather than explicit rewards. Our proposed algorithm consists of two main steps: (1) estimate the implicit reward using Maximum Likelihood Estimation (MLE) with general function approximation from offline data and (2) solve a distributionally robust planning problem over a confidence set around the MLE. We consider the general reward setting where the reward can be defined over the whole trajectory and provide a novel guarantee that allows us to learn any target policy with a polynomial number of samples, as long as the target policy is covered by the offline data. This guarantee is the first of its kind with general function approximation. To measure the coverage of the target policy, we introduce a new single-policy concentrability coefficient, which can be upper bounded by the per-trajectory concentrability coefficient. We also establish lower bounds that highlight the necessity of such concentrability and the difference from standard RL, where state-action-wise rewards are directly observed. We further extend and analyze our algorithm when the feedback is given over action pairs.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

39 of 53 references.

References · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  9. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  11. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  12. 2021Recursively Summarizing Books with Human Feedbackcited by 287
27 more
  1. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  4. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  7. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  8. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  9. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  10. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  11. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  12. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  13. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  14. 2019Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPscited by 104
  15. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  16. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  17. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 82
  18. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  19. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  20. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77
  21. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  22. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  23. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  24. 2020Fast Rates for Contextual Linear Optimizationcited by 43
  25. 2022PAC Reinforcement Learning for Predictive State Representationscited by 42
  26. 2021Fast Rates for the Regret of Offline Reinforcement Learningcited by 26
  27. 2021Gap-Dependent Unsupervised Exploration for Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 17; 2025: 22; 2026: 13.

55 papers.

Sort:

By area:

  1. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  2. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  3. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  4. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  5. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  6. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  7. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  8. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  9. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  10. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  11. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
  12. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspectivecited by 16
38 more
  1. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  2. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  3. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexitycited by 11
  4. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
  5. 2025Towards a Sharp Analysis of Offline Policy Learning for f-Divergence-Regularized Contextual Banditscited by 9
  6. 2023Demonstration-Regularized RLcited by 8
  7. 2024RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learningcited by 8
  8. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  9. 2024Exploiting Structure in Offline Multi-Agent RL: The Benefits of Low Interaction Rankcited by 6
  10. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  11. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  12. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  13. 2026Towards a Theoretical Understanding to the Generalization of RLHFcited by 5
  14. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspectivecited by 4
  15. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  16. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  17. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
  18. 2025Bayesian Optimization from Human Feedback: Near-Optimal Regret Boundscited by 3
  19. 2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
  20. 2025Improved Bounds for Private and Robust Alignmentcited by 3
  21. 2026Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluationscited by 3
  22. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
  23. 2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscited by 2
  24. 2025Lightweight Robust Direct Preference Optimizationcited by 2
  25. 2025Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Modelcited by 2
  26. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  27. 2026Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedbackcited by 2
  28. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learningcited by 1
  29. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  30. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiencycited by 0
  31. 2025Reinforcement Learning from Multi-level and Episodic Human Feedbackcited by 0
  32. 2025Learning Guarantee of Reward Modeling Using Deep Neural Networkscited by 0
  33. 2025Online Knowledge Distillation with Reward Guidancecited by 0
  34. 2025Thompson Sampling in Online RLHF with General Function Approximationcited by 0
  35. 2026Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Rewardcited by 0
  36. 2026Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distributioncited by 0
  37. 2026Generalisation of RLHF under Reward Shift and Clipped KL Regularisationcited by 0
  38. 2026Offline Constrained RLHF with Multiple Preference Oraclescited by 0

50 most cited of 55 All 55 (JSON)