mlsanityData to 14 Oct 2026

Learning Reward Functions by Integrating Human Demonstrations and Preferences

arXiv 1906.08928 ↗First posted 21 Jun 2019cs.RO cs.AIcited by 64

Not on the map. Why?

Abstract (arXiv)

Our goal is to accurately and efficiently learn reward functions for autonomous robots. Current approaches to this problem include inverse reinforcement learning (IRL), which uses expert demonstrations, and preference-based learning, which iteratively queries the user for her preferences between trajectories. In robotics however, IRL often struggles because it is difficult to get high-quality demonstrations; conversely, preference-based learning is very inefficient since it attempts to learn a continuous, high-dimensional function from binary feedback. We propose a new framework for reward learning, DemPref, that uses both demonstrations and preference queries to learn a reward function. Specifically, we (1) use the demonstrations to learn a coarse prior over the space of reward functions, to reduce the effective size of the space from which queries are generated; and (2) use the demonstrations to ground the (active) query generation process, to improve the quality of the generated queries. Our method alleviates the efficiency issues faced by standard preference-based learning methods and does not exclusively depend on (possibly low-quality) demonstrations. In numerical experiments, we find that DemPref is significantly more efficient than a standard active preference-based learning method. In a user study, we compare our method to a standard IRL method; we find that users rated the robot trained with DemPref as being more successful at learning their desired behavior, and preferred to use the DemPref system (over IRL) to train the robot.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 2; 2021: 13; 2022: 8; 2023: 11; 2024: 8; 2025: 12; 2026: 9.

64 papers.

Sort:

By area:

  1. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 114
  2. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 108
  3. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  4. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 71
  5. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 58
  6. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 56
  7. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 51
  8. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 49
  9. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  10. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31
  11. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  12. 2021Learning Reward Functions from Scale Feedbackcited by 24
38 more
  1. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  2. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignmentcited by 20
  3. 2025Learning from Active Human Involvement through Proxy Value Propagationcited by 19
  4. 2022A Ranking Game for Imitation Learningcited by 18
  5. 2021Offline Preference-Based Apprenticeship Learningcited by 17
  6. 2021APReL: A Library for Active Preference-based Reward Learning Algorithmscited by 14
  7. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  8. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  9. 2021Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Taskscited by 13
  10. 2023Active Reward Learning from Multiple Teacherscited by 13
  11. 2021The Reasonable Crowd: Towards evidence-based and interpretable models of driving behaviorcited by 12
  12. 2021Preference-based Learning of Reward Function Featurescited by 9
  13. 2021Preference learning along multiple criteria: A game-theoretic perspectivecited by 9
  14. 2023Toward Grounded Commonsense Reasoningcited by 9
  15. 2023CLIP-Motion: Learning Reward Functions for Robotic Actions Using Consecutive Observationscited by 9
  16. 2022Time-Efficient Reward Learning via Visually Assisted Cluster Rankingcited by 8
  17. 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
  18. 2019Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrationscited by 6
  19. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignmentcited by 4
  20. 2025Data-Efficient Learning from Human Interventions for Mobile Robotscited by 4
  21. 2021Cogment: Open Source Framework For Distributed Multi-actor Training, Deployment & Operationscited by 3
  22. 2023Learning Formal Specifications from Membership and Preference Queriescited by 3
  23. 2025Predictive Preference Learning from Human Interventionscited by 3
  24. 2026Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperationcited by 3
  25. 2021Similarity-Aware Skill Reproduction based on Multi-Representational Learning from Demonstrationcited by 2
  26. 2023Towards Regulatable AI Systems: Technical Gaps and Policy Opportunitiescited by 2
  27. 2023Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learningcited by 2
  28. 2024How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robotscited by 2
  29. 2025Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratchcited by 2
  30. 2025Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Boundscited by 2
  31. 2025From Learning to Mastery: Achieving Safe and Efficient Real-World Autonomous Driving with Human-In-The-Loop Reinforcement Learningcited by 2
  32. 2026Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulationcited by 2
  33. 2022Strategy Discovery and Mixture in Lifelong Learning from Heterogeneous Demonstrationcited by 1
  34. 2022A Survey on Deep Reinforcement Learning-based Approaches for Adaptation and Generalizationcited by 1
  35. 2023Genetic Imitation Learning by Reward Extrapolationcited by 1
  36. 2023A Markovian Formalism for Active Queryingcited by 1
  37. 2023Active Inverse Learning in Stackelberg Trajectory Gamescited by 1
  38. 2024On the Exponential Convergence for Offline RLHF with Pairwise Comparisonscited by 1

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Learning Preferences for Interactive Autonomy
  2. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  3. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  4. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regression
  5. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
  6. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
  7. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  8. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedback