mlsanityData to 14 Oct 2026

Rethinking the Implications of Human Feedback for Preference Learning in Human-Robot Collaboration

arXiv 2609.13982 ↗First posted 12 Sep 2026cs.RO cs.AI cs.HCcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

In Human-Robot Interaction, the standard approach to learn a reward model that represents human preferences for robot behavior consists of three steps. First, the robot collects limited direct evidence from human feedback (e.g., positive or negative binary feedback). Then, the robot utilizes the direct evidence to derive accepted or rejected labels to feasible but unchosen actions using fixed implication rules. Finally, the robot updates the reward model with both the direct and derived evidence. Unfortunately, the fixed rule can hinder preference learning: in a user study with two collaborative simulation environments, human-provided implication labels often differed from the standard fixed rule, and using the human labels substantially improved reward learning with the Preference Learning from Implicit and Explicit Feedback (PIE) algorithm. Consequently, we propose IMPLIED, an implication modeling method that treats fixed-rule implications as an initial guide while learning to infer and revise accepted and rejected action labels over time. Across evaluations on recorded human-robot interaction trajectories and a physical robot pizza-making study, IMPLIED predicts human implications more accurately than the fixed rule approach and LLM baselines, approaching the performance of a human-label oracle. In turn, IMPLIED reduces preference-estimation error and leads to robot actions that are more often rational with respect to a combined reward (which includes the true preference reward and a task-specific reward) compared to baselines. By learning to reason about the implications of human feedback, this work enables more faithful and efficient robot behavior adaptation during human-robot collaboration.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 51 references that are in the map. First posted on arXiv 12 Sep 2026.

Builds on

30 of 51 references.

References · 30

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  3. 2016Cooperative Inverse Reinforcement Learningcited by 300
  4. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  5. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  6. 2016GPflow: A Gaussian process library using TensorFlowcited by 140
  7. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  8. 2017Behavior Trees in Robotics and AI: An Introductioncited by 107
  9. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  10. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  11. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  12. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
18 more
  1. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedbackcited by 38
  2. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 35
  3. 2024Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguitycited by 27
  4. 2016Introspective Perception: Learning to Predict Failures in Vision Systemscited by 17
  5. 2018Including Uncertainty when Learning from Human Correctionscited by 15
  6. 2020Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approachcited by 10
  7. 2019Improving User Specifications for Robot Behavior through Active Preference Learning: Framework and Evaluationcited by 7
  8. 2025Open-Ended Goal Inference through Actions and Language for Human-Robot Collaborationcited by 5
  9. 2024Goal Inference from Open-Ended Dialogcited by 4
  10. 2025Effects of Robot Competency and Motion Legibility on Human Correction Feedbackcited by 4
  11. 2026Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimizationcited by 4
  12. 2024REACT: Two Datasets for Analyzing Both Human Reactions and Evaluative Feedback to Robots Over Timecited by 3
  13. 2025Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitationcited by 3
  14. 2025I've Changed My Mind: Robots Adapting to Changing Human Goals during Collaborationcited by 3
  15. 2023Predicting Human Perceptions of Robot Performance During Navigation Taskscited by 2
  16. 2026InterPReT: Interactive Policy Restructuring and Training Enable Effective Imitation Learning from Laypersonscited by 2
  17. 2026Enhancing Goal Inference via Correction Timingcited by 1
  18. 2026Robots Influencing Humans to Reveal their Goals during Collaboration and Competitioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  2. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction
  3. 2025Influencing Humans to Conform to Preference Models for RLHF
  4. 2024PrefMMT: Modeling Human Preferences in Preference-based Reinforcement Learning with Multimodal Transformers
  5. 2025Predictive Preference Learning from Human Interventions
  6. 2023Effect of Adapting to Human Preferences on Trust in Human-Robot Teaming ↗
  7. 2026Freeform Preference Learning for Robotic Manipulation
  8. 2022Feedback-efficient Active Preference Learning for Socially Aware Robot Navigation