mlsanityData to 7 Oct 2026

The EMPATHIC Framework for Task Learning from Implicit Human Feedback

arXiv 2009.13649 ↗First posted 28 Sep 2020cs.HC cs.ROcited by 38

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reactions such as gestures, facial expressions, and vocalizations are an abundant, naturally occurring channel of information that humans provide during interactions. A robot or other agent could leverage an understanding of such implicit human feedback to improve its task performance at no cost to the human. This approach contrasts with common agent teaching methods based on demonstrations, critiques, or other guidance that need to be attentively and intentionally provided. In this paper, we first define the general problem of learning from implicit human feedback and then propose to address this problem through a novel data-driven framework, EMPATHIC. This two-stage method consists of (1) mapping implicit human feedback to relevant task statistics such as reward, optimality, and advantage; and (2) using such a mapping to learn a task. We instantiate the first stage and three second-stage evaluations of the learned mapping. To do so, we collect a dataset of human facial reactions while participants observe an agent execute a sub-optimal policy for a prescribed training task. We train a deep neural network on this data and demonstrate its ability to (1) infer relative reward ranking of events in the training task from prerecorded human facial reactions; (2) improve the policy of an agent in the training task using live human facial reactions; and (3) transfer to a novel domain in which it evaluates robot manipulation trajectories.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 48 references that are in the map. First posted on arXiv 28 Sep 2020.

Builds on

12 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback28 Oct

Other references · 11

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Deep Facial Expression Recognition: A Surveycited by 269
  3. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  4. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 136
  5. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  6. 2019Leveraging Human Guidance for Deep Reinforcement Learning Taskscited by 42
  7. 2017Objective Classes for Micro-Facial Expression Recognitioncited by 14
  8. 2016Face valuing: Training user interfaces with facial expressions and reinforcement learningcited by 6
  9. 2018Learning via social awareness: Improving a deep generative sketching model with facial feedbackcited by 3
  10. 2019Modeling Affect-based Intrinsic Rewards for Exploration and Learningcited by 1
  11. 2020Facial Feedback for Reinforcement Learning: A Case Study and Offline Analysis Using the TAMER Frameworkcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 5; 2022: 8; 2023: 6; 2024: 6; 2025: 9; 2026: 3.

38 papers.

Sort:

By area:

  1. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  2. 2022Models of human preference for learning reward functionscited by 54
  3. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  4. 2024UMBRAE: Unified Multimodal Brain Decodingcited by 31
  5. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  6. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  7. 2023CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videoscited by 14
  8. 2021Conflict Avoidance in Social Navigation -- a Surveycited by 13
  9. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  10. 2022First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximizationcited by 8
  11. 2023NOIR: Neural Signal Operated Intelligent Robots for Everyday Activitiescited by 8
  12. 2022Interaction-Grounded Learning with Action-inclusive Feedbackcited by 7
26 more
  1. 2023From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learningcited by 6
  2. 2021Assisted Robust Reward Designcited by 5
  3. 2022X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedbackcited by 4
  4. 2022Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfacescited by 4
  5. 2024Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaborationcited by 4
  6. 2021Improving Reinforcement Learning with Human Assistance: An Argument for Human Subject Studies with HIPPO Gymcited by 3
  7. 2025Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitationcited by 3
  8. 2022Multi-trainer Interactive Reinforcement Learning Systemcited by 2
  9. 2023Predicting Human Perceptions of Robot Performance During Navigation Taskscited by 2
  10. 2024How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robotscited by 2
  11. 2024Robot Behavior Personalization from Sparse User Feedbackcited by 2
  12. 2021A MultiModal Social Robot Toward Personalized Emotion Interactioncited by 1
  13. 2023Teach Me How to Learn: A Perspective Review towards User-centered Neuro-symbolic Learning for Robotic Surgical Systemscited by 1
  14. 2025The Social Context of Human-Robot Interactionscited by 1
  15. 2025Towards Reinforcement Learning from Neural Feedback: Mapping fNIRS Signals to Agent Performancecited by 1
  16. 2023Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systemscited by 0
  17. 2024Prosody as a Teaching Signal for Agent Learning: Exploratory Studies and Algorithmic Implicationscited by 0
  18. 2025NatSGLD: A Dataset with Speech, Gesture, Logic, and Demonstration for Robot Learning in Natural Human-Robot Interactioncited by 0
  19. 2025Supervised Reward Inferencecited by 0
  20. 2025Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approachcited by 0
  21. 2025QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agentscited by 0
  22. 2025Accelerating Reinforcement Learning via Error-Related Human Brain Signalscited by 0
  23. 2025Reinforcement Learning from Implicit Neural Feedback for Human-Aligned Robot Controlcited by 0
  24. 2026An offline approach to fNIRS-guided reinforcement learning for robot behaviorcited by 0
  25. 2026Rethinking the Implications of Human Feedback for Preference Learning in Human-Robot Collaborationcited by 0
  26. 2026PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method
  2. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  3. 2022Teaching Drones on the Fly: Can Emotional Feedback Serve as Learning Signal for Training Artificial Agents?
  4. 2025Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback
  5. 2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback
  6. 2025PEER: Unified Process-Outcome Reinforcement Learning for Structured Empathetic Reasoning
  7. 2026Facial Expression Generation Aligned with Human Preference for Natural Dyadic Interaction
  8. 2024Empathic Grounding: Explorations using Multimodal Interaction and Large Language Models with Conversational Agents