mlsanityData to 7 Oct 2026

Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences

arXiv 2006.14091 ↗First posted 24 Jun 2020cs.RO cs.AI cs.LGcited by 71

Not on the map. Why?

Cited by 24 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reward functions are a common way to specify the objective of a robot. As designing reward functions can be extremely challenging, a more promising approach is to directly learn reward functions from human teachers. Importantly, data from human teachers can be collected either passively or actively in a variety of forms: passive data sources include demonstrations, (e.g., kinesthetic guidance), whereas preferences (e.g., comparative rankings) are actively elicited. Prior research has independently applied reward learning to these different data sources. However, there exist many domains where multiple sources are complementary and expressive. Motivated by this general problem, we present a framework to integrate multiple sources of information, which are either passively or actively collected from human users. In particular, we present an algorithm that first utilizes user demonstrations to initialize a belief about the reward function, and then actively probes the user with preference queries to zero-in on their true reward. This algorithm not only enables us combine multiple data sources, but it also informs the robot when it should leverage each type of information. Further, our approach accounts for the human's ability to provide data: yielding user-friendly preference queries which are also theoretically optimal. Our extensive simulated experiments and user studies on a Fetch mobile manipulator demonstrate the superiority and the usability of our integrated framework.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 15; 2024: 23; 2025: 15; 2026: 4.

71 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  3. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  5. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  6. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  7. 2022Models of human preference for learning reward functionscited by 54
  8. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  9. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  10. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  11. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  12. 2021Learning Multimodal Rewards from Rankingscited by 41
38 more
  1. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  2. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  3. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedbackcited by 25
  4. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
  5. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignmentcited by 19
  6. 2022A Ranking Game for Imitation Learningcited by 18
  7. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 18
  8. 2023Learning Optimal Advantage from Preferences and Mistaking it for Rewardcited by 18
  9. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedbackcited by 15
  10. 2024Fine-Tuning Language Models with Reward Learning on Policycited by 15
  11. 2021APReL: A Library for Active Preference-based Reward Learning Algorithmscited by 14
  12. 2023Rating-based Reinforcement Learningcited by 14
  13. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  14. 2025LAMARL: LLM-Aided Multi-Agent Reinforcement Learning for Cooperative Policy Generationcited by 13
  15. 2023Active Reward Learning from Online Preferencescited by 12
  16. 2024Learning to Assist Humans without Inferring Rewardscited by 12
  17. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  18. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10
  19. 2023Toward Grounded Commonsense Reasoningcited by 9
  20. 2025CREStE: Scalable Mapless Navigation with Internet Scale Priors and Counterfactual Guidancecited by 9
  21. 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
  22. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Modelscited by 7
  23. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  24. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 5
  25. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignmentcited by 4
  26. 2025Influencing Humans to Conform to Preference Models for RLHFcited by 4
  27. 2022Learning Performance Graphs from Demonstrations via Task-Based Evaluationscited by 3
  28. 2025Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitationcited by 3
  29. 2021GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedbackcited by 2
  30. 2023Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learningcited by 2
  31. 2024The Role of Higher-Order Cognitive Models in Active Learningcited by 2
  32. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferencescited by 2
  33. 2025Optimal Interactive Learning on the Job via Facility Location Planningcited by 2
  34. 2022Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learningcited by 1
  35. 2023Policy Evaluation in Decentralized POMDPs with Belief Sharingcited by 1
  36. 2023A Markovian Formalism for Active Queryingcited by 1
  37. 2024A Generalized Acquisition Function for Preference-based Reward Learningcited by 1
  38. 2024MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Interventioncited by 1

50 most cited of 71 All 71 (JSON)