mlsanityData to 7 Oct 2026

Advances in Preference-based Reinforcement Learning: A Review

arXiv 2408.11943 ↗First posted 21 Aug 2024cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) algorithms suffer from the dependency on accurately engineered reward functions to properly guide the learning agents to do the required tasks. Preference-based reinforcement learning (PbRL) addresses that by utilizing human preferences as feedback from the experts instead of numeric rewards. Due to its promising advantage over traditional RL, PbRL has gained more focus in recent years with many significant advances. In this survey, we present a unified PbRL framework to include the newly emerging approaches that improve the scalability and efficiency of PbRL. In addition, we give a detailed overview of the theoretical guarantees and benchmarking work done in the field, while presenting its recent applications in complex real-world tasks. Lastly, we go over the limitations of the current approaches and the proposed future research directions.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 58 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

29 of 58 references.

References · 29

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2018DeepMind Control Suitecited by 935
  11. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  12. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
17 more
  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  2. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  3. 2013Learning to Optimize Via Posterior Samplingcited by 253
  4. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  5. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  6. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  7. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  8. 2020Improved Optimistic Algorithms for Logistic Banditscited by 117
  9. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  10. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  11. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  12. 2019A Survey of Deep Reinforcement Learning in Video Gamescited by 73
  13. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  14. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  15. 2017Maximum Selection and Ranking under Noisy Comparisonscited by 18
  16. 2019Preference-based Interactive Multi-Document Summarisationcited by 11
  17. 2018APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  2. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  3. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 1
  4. 2026RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedbackcited by 1
  5. 2026Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparabilitycited by 0
  6. 2026Enhancing VLM Reward Models Through Structure-Aware Fine-Tuningcited by 0