mlsanityData to 7 Oct 2026

Dueling RL: Reinforcement Learning with Trajectory Preferences

arXiv 2111.04850 ↗First posted 8 Nov 2021cs.LGcited by 105

Not on the map. Why?

Cited by 70 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We consider the problem of preference based reinforcement learning (PbRL), where, unlike traditional reinforcement learning, an agent receives feedback only in terms of a 1 bit (0/1) preference over a trajectory pair instead of absolute rewards for them. The success of the traditional RL framework crucially relies on the underlying agent-reward model, which, however, depends on how accurately a system designer can express an appropriate reward function and often a non-trivial task. The main novelty of our framework is the ability to learn from preference-based trajectory feedback that eliminates the need to hand-craft numeric reward models. This paper sets up a formal framework for the PbRL problem with non-markovian rewards, where the trajectory preferences are encoded by a generalized linear model of dimension d. Assuming the transition model is known, we then propose an algorithm with almost optimal regret guarantee of tilde Oleft( SH d log (T / δ) √T right). We further, extend the above algorithm to the case of unknown transition dynamics, and provide an algorithm with near optimal regret guarantee widetildeO((√d + H2 + |S|)√dT +√|S||A|TH ). To the best of our knowledge, our work is one of the first to give tight regret guarantees for preference based RL problems with trajectory preferences.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Builds on 33 of its 65 references that are in the map. First posted on arXiv 8 Nov 2021.

Builds on

33 of 65 references.

References · 33

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  4. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  5. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 220
  6. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  7. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
  8. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  9. 2020Improved Optimistic Algorithms for Logistic Banditscited by 117
  10. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  11. 2020Provable Self-Play Algorithms for Competitive Reinforcement Learningcited by 112
  12. 2020Near-Optimal Reinforcement Learning with Self-Playcited by 108
21 more
  1. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 105
  2. 2020A Sharp Analysis of Model-based Reinforcement Learning with Self-Playcited by 104
  3. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 104
  4. 2020Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibriumcited by 100
  5. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  6. 2014Reducing Dueling Bandits to Cardinal Banditscited by 62
  7. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  8. 2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 51
  9. 2019Regret Minimization for Reinforcement Learning by Evaluating the Optimal Bias Functioncited by 43
  10. 2018Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in MDPscited by 41
  11. 2020Reinforcement Learning with Trajectory Feedbackcited by 40
  12. 2017Provably Optimal Algorithms for Generalized Linear Contextual Banditscited by 38
  13. 2015Copeland Dueling Banditscited by 36
  14. 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
  15. 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
  16. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  17. 2016A Relative Exponential Weighing Algorithm for Adversarial Utility-based Dueling Banditscited by 31
  18. 2018PAC Battling Bandits in the Plackett-Luce Modelcited by 27
  19. 2018Active Ranking with Subset-wise Preferencescited by 21
  20. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  21. 2018Regret Bounds for Reinforcement Learning via Markov Chain Concentrationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 25; 2024: 36; 2025: 29; 2026: 13.

105 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  4. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  6. 2023Nash Learning from Human Feedbackcited by 220
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  8. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
  9. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  10. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  11. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  12. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
38 more
  1. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  2. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  3. 2023Is RLHF More Difficult than Standard RL?cited by 96
  4. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  5. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  6. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  7. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  8. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  9. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  10. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  11. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  12. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  13. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  14. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  15. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  16. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
  17. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  18. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  19. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  20. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  21. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  22. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
  23. 2024RLVF: Learning from Verbal Feedback without Overgeneralizationcited by 18
  24. 2024Optimal Design for Human Preference Elicitationcited by 17
  25. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
  26. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspectivecited by 16
  27. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  28. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problemscited by 12
  29. 2023Differentially Private Reward Estimation with Preference Feedbackcited by 12
  30. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  31. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  32. 2024Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedbackcited by 10
  33. 2024Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexitycited by 10
  34. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuningcited by 10
  35. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  36. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 10
  37. 2023When is Agnostic Reinforcement Learning Statistically Tractable?cited by 8
  38. 2024Almost Free: Self-concordance in Natural Exponential Families and an Application to Banditscited by 8

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Direct Preference-based Policy Optimization without Reward Modeling
  2. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  3. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  4. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  5. 2026SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
  6. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  7. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  8. 2024Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs