mlsanityData to 7 Oct 2026

Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation

arXiv 2205.11140 ↗First posted 23 May 2022cs.LGcited by 80

Not on the map. Why?

Cited by 61 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study human-in-the-loop reinforcement learning (RL) with trajectory preferences, where instead of receiving a numeric reward at each step, the agent only receives preferences over trajectory pairs from a human overseer. The goal of the agent is to learn the optimal policy which is most preferred by the human overseer. Despite the empirical successes, the theoretical understanding of preference-based RL (PbRL) is only limited to the tabular case. In this paper, we propose the first optimistic model-based algorithm for PbRL with general function approximation, which estimates the model using value-targeted regression and calculates the exploratory policies by solving an optimistic planning problem. Our algorithm achieves the regret of tildeO (operatornamepoly(d H) √K ), where d is the complexity measure of the transition and preference model depending on the Eluder dimension and log-covering numbers, H is the planning horizon, K is the number of episodes, and tilde O(·) omits logarithmic terms. Our lower bound indicates that our algorithm is near-optimal when specialized to the linear setting. Furthermore, we extend the PbRL problem by formulating a novel problem called RL with n-wise comparisons, and provide the first sample-efficient algorithm for this new setting. To the best of our knowledge, this is the first theoretical result for PbRL with (general) function approximation.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

34 of 54 references.

References · 34

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  6. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  7. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  8. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  9. 2013Learning to Optimize Via Posterior Samplingcited by 253
  10. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  11. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  12. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
22 more
  1. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  2. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  3. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  4. 2016Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Roboticscited by 134
  5. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  6. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  7. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  8. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
  9. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  10. 2020Instance-Dependent Complexity of Contextual Bandits and Reinforcement Learning: A Disagreement-Based Perspectivecited by 78
  11. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 69
  12. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  13. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  14. 2020Reinforcement Learning with Trajectory Feedbackcited by 40
  15. 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
  16. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  17. 2016Learning Preferences for Manipulation Tasks from Online Coactive Feedbackcited by 31
  18. 2021Reward-Free Model-Based Reinforcement Learning with Linear Function Approximationcited by 29
  19. 2017Maximum Selection and Ranking under Noisy Comparisonscited by 18
  20. 2021Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solvercited by 13
  21. 2017Learning from Comparisons and Choicescited by 12
  22. 2019Zeroth Order Non-convex optimization with Dueling-Choice Banditscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 31; 2025: 25; 2026: 11.

80 papers.

Sort:

By area:

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  4. 2023Nash Learning from Human Feedbackcited by 220
  5. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  6. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  7. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  8. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  9. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  10. 2023Is RLHF More Difficult than Standard RL?cited by 96
  11. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  12. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
38 more
  1. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  2. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  3. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  4. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  5. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  6. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  7. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  8. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  9. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  10. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  11. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  12. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  13. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
  14. 2024Optimal Design for Human Preference Elicitationcited by 17
  15. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
  16. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  17. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problemscited by 12
  18. 2023Differentially Private Reward Estimation with Preference Feedbackcited by 12
  19. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  20. 2024Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedbackcited by 10
  21. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuningcited by 10
  22. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  23. 2024RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learningcited by 8
  24. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  25. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  26. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  27. 2025Contextual Online Uncertainty-Aware Preference Learning for Human Feedbackcited by 7
  28. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  29. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  30. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
  31. 2026Towards a Theoretical Understanding to the Generalization of RLHFcited by 5
  32. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysiscited by 4
  33. 2024RL-STaR: Theoretical Analysis of Reinforcement Learning Frameworks for Self-Taught Reasonercited by 4
  34. 2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Samplingcited by 4
  35. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  36. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
  37. 2024Preference Elicitation for Offline Reinforcement Learningcited by 3
  38. 2025Bayesian Optimization from Human Feedback: Near-Optimal Regret Boundscited by 3

50 most cited of 80 All 80 (JSON)