mlsanityData to 7 Oct 2026

Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models

arXiv 2301.04741 ↗First posted 11 Jan 2023cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) can enable robots to learn to perform tasks based on an individual's preferences without requiring a hand-crafted reward function. However, existing approaches either assume access to a high-fidelity simulator or analytic model or take a model-free approach that requires extensive, possibly unsafe online environment interactions. In this paper, we study the benefits and challenges of using a learned dynamics model when performing PbRL. In particular, we provide evidence that a learned dynamics model offers the following benefits when performing PbRL: (1) preference elicitation and policy optimization require significantly fewer environment interactions than model-free PbRL, (2) diverse preference queries can be synthesized safely and efficiently as a byproduct of standard model-based RL, and (3) reward pre-training based on suboptimal demonstrations can be performed without any environmental interaction. Our paper provides empirical evidence that learned dynamics models enable robots to learn customized policies based on user preferences in ways that are safer and more sample efficient than prior preference learning approaches. Supplementary materials and code are available at https://sites.google.com/berkeley.edu/mop-rl.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 41 of its 68 references that are in the map. First posted on arXiv 11 Jan 2023.

Builds on

41 of 68 references.

References · 41

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2016OpenAI Gymcited by 1,684
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  9. 2019Model-Based Reinforcement Learning for Ataricited by 495
  10. 2018Behavioral Cloning from Observationcited by 482
  11. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
29 more
  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  2. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  3. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  4. 2018Stochastic Video Generation with a Learned Priorcited by 244
  5. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  6. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  7. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  8. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  9. 2020Learning Stable Deep Dynamics Modelscited by 110
  10. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  11. 2018Imitating Latent Policies from Observationcited by 101
  12. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  13. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  14. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  15. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  16. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  17. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  18. 2019Assistive Gym: A Physics Simulation Framework for Assistive Roboticscited by 44
  19. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  20. 2019Learning Human Objectives by Evaluating Hypothetical Behaviorcited by 41
  21. 2019Variational Inference MPC for Bayesian Model-based Reinforcement Learningcited by 39
  22. 2016Learning Preferences for Manipulation Tasks from Online Coactive Feedbackcited by 31
  23. 2016Model-based Adversarial Imitation Learningcited by 29
  24. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  25. 2021Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Taskscited by 12
  26. 2019Curious iLQR: Resolving Uncertainty in Model-based RLcited by 11
  27. 2019Model-based Behavioral Cloning with Future Image Similarity Learningcited by 8
  28. 2019Asynchronous Methods for Model-Based Reinforcement Learningcited by 7
  29. 2021Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Interventioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 3; 2026: 6.

20 papers.

Sort:

By area:

  1. 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
  2. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  3. 2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learningcited by 4
  4. 2024Preference Elicitation for Offline Reinforcement Learningcited by 3
  5. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 3
  6. 2024Scalable Interactive Machine Learning for Future Command and Controlcited by 2
  7. 2024Reward Learning from Suboptimal Demonstrations with Applications in Surgical Electrocauterycited by 2
  8. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learningcited by 1
  9. 2023Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?cited by 0
  10. 2024Integrating Human Expertise in Continuous Spaces: A Novel Interactive Bayesian Optimization Framework with Preference Expected Improvementcited by 0
  11. 2024Preference-Guided Reinforcement Learning for Efficient Explorationcited by 0
  12. 2024Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrationscited by 0
8 more
  1. 2024Effective Reward Specification in Deep Reinforcement Learningcited by 0
  2. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learningcited by 0
  3. 2026Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformerscited by 0
  4. 2026Implicit Safety Alignment from Crowd Preferencescited by 0
  5. 2026UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learningcited by 0
  6. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learningcited by 0
  7. 2026Learning Safe Agent Behaviour from Human Preferences and Justifications via World Modelscited by 0
  8. 2026Specifying Reward Functions for RL Without Environment Samplingcited by 0