mlsanityData to 7 Oct 2026

Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability

arXiv 2607.11432 ↗First posted 13 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this work, we study the reinforcement learning (RL) problem from pairwise trajectory comparisons provided by a human expert. We generalize preference-based RL by formalizing a novel setting in which the expert can also label trajectory pairs as incomparable, i.e., when neither trajectory dominates the other. We introduce the learning problem and the desiderata that its solution should satisfy. Then, we propose a novel Bradley-Terry-inspired rationality model that effectively captures incomparabilities and infers a multi-dimensional reward function, and we study its properties. We provide a sample complexity analysis for learning the model parameters when a dataset is available. Finally, we evaluate our model's ability to reconstruct a reward function that aligns with the expert's comparisons in simulated environments and to recover the Pareto frontier of policies, along with a robustness analysis across varying levels of expert rationality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 65 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

33 of 65 references.

References · 33

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2021Convex optimizationcited by 2,479
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  12. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
21 more
  1. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  2. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  3. 2023Nash Learning from Human Feedbackcited by 220
  4. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  5. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  6. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  7. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  8. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  9. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  10. 2021Machine Learning with a Reject Option: A surveycited by 107
  11. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  12. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  13. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  14. 2021Reward-Free Model-Based Reinforcement Learning with Linear Function Approximationcited by 29
  15. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24
  16. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learningcited by 13
  17. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9
  18. 2024RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learningcited by 8
  19. 2025Preference-based Multi-Objective Reinforcement Learningcited by 7
  20. 2024Advances in Preference-based Reinforcement Learning: A Reviewcited by 6
  21. 2026A Reward-Free Viewpoint on Multi-Objective Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Dueling RL: Reinforcement Learning with Trajectory Preferences
  2. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
  3. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  4. 2026UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning
  5. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences
  7. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  8. 2025Reward Compatibility: A Framework for Inverse RL