mlsanityData to 7 Oct 2026

B-Pref: Benchmarking Preference-Based Reinforcement Learning

arXiv 2111.03026 ↗First posted 4 Nov 2021cs.LG cs.AI cs.HCcited by 106

Not on the map. Why?

Cited by 63 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning (RL) requires access to a reward function that incentivizes the right behavior, but these are notoriously hard to specify for complex tasks. Preference-based RL provides an alternative: learning policies using a teacher's preferences without pre-defined rewards, thus overcoming concerns associated with reward engineering. However, it is difficult to quantify the progress in preference-based RL due to the lack of a commonly adopted benchmark. In this paper, we introduce B-Pref: a benchmark specially designed for preference-based RL. A key challenge with such a benchmark is providing the ability to evaluate candidate algorithms quickly, which makes relying on real human input for evaluation prohibitive. At the same time, simulating human input as giving perfect preferences for the ground truth reward function is unrealistic. B-Pref alleviates this by simulating teachers with a wide array of irrationalities, and proposes metrics not solely for performance but also for robustness to these potential irrationalities. We showcase the utility of B-Pref by using it to analyze algorithmic design choices, such as selecting informative queries, for state-of-the-art preference-based RL algorithms. We hope that B-Pref can serve as a common starting point to study preference-based RL more systematically. Source code is available at https://github.com/rll-research/B-Pref.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 51 of its 89 references that are in the map. First posted on arXiv 4 Nov 2021.

Builds on

51 of 89 references.

References · 51

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2016Concrete Problems in AI Safetycited by 1,582
38 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  2. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  4. 2018DeepMind Control Suitecited by 935
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2017Deep Reinforcement Learning that Matterscited by 727
  9. 2018Learning Dexterous In-Hand Manipulationcited by 643
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  11. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  12. 2020Reinforcement Learning with Augmented Datacited by 452
  13. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  14. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  15. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  16. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  17. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  18. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  19. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  20. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  21. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  22. 2017Inverse Reward Designcited by 246
  23. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  24. 2019Efficient Exploration via State Marginal Matchingcited by 186
  25. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  26. 2018Meta-Gradient Reinforcement Learningcited by 156
  27. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  28. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  29. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  30. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  31. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  32. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  33. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  34. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  35. 2019Learning Human Objectives by Evaluating Hypothetical Behaviorcited by 41
  36. 2019Preferences Implicit in the State of the Worldcited by 33
  37. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  38. 2020Avoiding Side Effects in Complex Environmentscited by 24

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 21; 2024: 34; 2025: 20; 2026: 19.

106 papers.

Sort:

By area:

  1. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  2. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  3. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  4. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  5. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  6. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  7. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  8. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  9. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  10. 2022Models of human preference for learning reward functionscited by 54
  11. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 52
  12. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
38 more
  1. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 39
  2. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  3. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  4. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  5. 2023Reinforcement Learning from Diverse Human Preferencescited by 24
  6. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  7. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  8. 2023On The Fragility of Learned Reward Functionscited by 21
  9. 2022The Expertise Problem: Learning from Specialized Feedbackcited by 20
  10. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
  11. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  12. 2023Learning Optimal Advantage from Preferences and Mistaking it for Rewardcited by 18
  13. 2024Optimal Design for Human Preference Elicitationcited by 17
  14. 2022Transformers are Adaptable Task Plannerscited by 15
  15. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  16. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  17. 2023Rating-based Reinforcement Learningcited by 14
  18. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  19. 2022Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learningcited by 13
  20. 2023Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedbackcited by 12
  21. 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
  22. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
  23. 2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
  24. 2025Speaking the Language of Teamwork: LLM-Guided Credit Assignment in Multi-Agent Reinforcement Learningcited by 10
  25. 2025On The Sample Complexity Bounds In Bilevel Reinforcement Learningcited by 10
  26. 2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulationcited by 9
  27. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewardscited by 9
  28. 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
  29. 2024Pareto-Optimal Learning from Preferences with Hidden Contextcited by 8
  30. 2023Maximum Causal Entropy Inverse Constrained Reinforcement Learningcited by 7
  31. 2024Crowd-PrefRL: Preference-Based Reward Learning from Crowdscited by 7
  32. 2024ICPL: Few-shot In-context Preference Learning via LLMscited by 7
  33. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Modelscited by 7
  34. 2024Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Modelcited by 7
  35. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrationscited by 7
  36. 2025Preference-based Multi-Objective Reinforcement Learningcited by 7
  37. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  38. 2024Advances in Preference-based Reinforcement Learning: A Reviewcited by 6

50 most cited of 106 All 106 (JSON)