Data to 7 Oct 2026

Preference-based Reinforcement Learning with Finite-Time Guarantees

arXiv 2006.08910 ↗First posted 16 Jun 2020cs.LG cs.AI stat.MLcited by 65

Not on the map. Why?

Cited by 39 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Preference-based Reinforcement Learning (PbRL) replaces reward values in traditional reinforcement learning by preferences to better elicit human opinion on the target objective, especially when numerical reward values are hard to design or interpret. Despite promising results in applications, the theoretical understanding of PbRL is still in its infancy. In this paper, we present the first finite-time analysis for general PbRL problems. We first show that a unique optimal policy may not exist if preferences over trajectories are deterministic for PbRL. If preferences are stochastic, and the preference probability relates to the hidden reward values, we present algorithms for PbRL, both with and without a simulator, that are able to identify the best policy up to accuracy varepsilon with high probability. Our method explores the state space by navigating to under-explored states, and solves PbRL using a combination of dueling bandits and policy search. Experiments show the efficacy of our method when it is applied to real-world problems.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 2; 2023: 12; 2024: 16; 2025: 19; 2026: 12.

65 papers.

Sort:

By area:

2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
2023Is RLHF More Difficult than Standard RL?cited by 96
2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
2021On the Expressivity of Markov Rewardcited by 71
2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
2023Provable Offline Preference-Based Reinforcement Learningcited by 55
2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46

38 more

2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
2024Dataset Reset Policy Optimization for RLHFcited by 41
2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
2024Optimal Design for Human Preference Elicitationcited by 17
2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
2023Rating-based Reinforcement Learningcited by 14
2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problemscited by 12
2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
2021Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functionscited by 10
2025On The Sample Complexity Bounds In Bilevel Reinforcement Learningcited by 10
2024RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learningcited by 8
2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
2024Advances in Preference-based Reinforcement Learning: A Reviewcited by 6
2024Conversational Dueling Bandits in Generalized Linear Modelscited by 5
2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
2026Towards a Theoretical Understanding to the Generalization of RLHFcited by 5
2022Reinforcement Learning with a Terminatorcited by 4
2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspectivecited by 4
2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysiscited by 4
2025Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Samplingcited by 4
2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
2023Conditions on Preference Relations that Guarantee the Existence of Optimal Policiescited by 3
2025Bayesian Optimization from Human Feedback: Near-Optimal Regret Boundscited by 3
2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3
2025On the Role of Preference Variance in Preference Optimizationcited by 3
2025Improved Bounds for Private and Robust Alignmentcited by 3
2025Leveraging Sparsity for Sample-Efficient Preference Learning: A Theoretical Perspectivecited by 2
2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learningcited by 2
2025Reinforcement Learning from Adversarial Preferences in Tabular MDPscited by 2
2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscited by 2
2026Learning Dynamics in RL Post-Training for Language Modelscited by 2

50 most cited of 65 All 65 (JSON)