# The EMPATHIC Framework for Task Learning from Implicit Human Feedback

arXiv: https://arxiv.org/abs/2009.13649 · HTML: https://mlsanity.com/paper/2009.13649/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 12 of its 48 references that are in the map.
- First posted on arXiv 28 Sep 2020.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Leveraging Human Guidance for Deep Reinforcement Learning Tasks](https://mlsanity.com/paper/1909.09906/) (2019)
- [DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback](https://mlsanity.com/paper/1810.11748/) (2018)
- [Objective Classes for Micro-Facial Expression Recognition](https://mlsanity.com/paper/1708.07549/) (2017)
- [Face valuing: Training user interfaces with facial expressions and reinforcement learning](https://mlsanity.com/paper/1606.02807/) (2016)
- [Learning via social awareness: Improving a deep generative sketching model with facial feedback](https://mlsanity.com/paper/1802.04877/) (2018)
- [Modeling Affect-based Intrinsic Rewards for Exploration and Learning](https://mlsanity.com/paper/1912.00403/) (2019)
- [Facial Feedback for Reinforcement Learning: A Case Study and Offline Analysis Using the TAMER Framework](https://mlsanity.com/paper/2001.08703/) (2020)

## Built on by (most cited first, 30 of 38)

- [Language-Conditioned Imitation Learning for Robot Manipulation Tasks](https://mlsanity.com/paper/2010.12083/) (2020)
- [Models of human preference for learning reward functions](https://mlsanity.com/paper/2206.02231/) (2022)
- [AI Alignment with Changing and Influenceable Reward Functions](https://mlsanity.com/paper/2405.17713/) (2024)
- [UMBRAE: Unified Multimodal Brain Decoding](https://mlsanity.com/paper/2404.07202/) (2024)
- [Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning](https://mlsanity.com/paper/2206.13316/) (2022)
- [Continual Learning for Instruction Following from Realtime Feedback](https://mlsanity.com/paper/2212.09710/) (2022)
- [Conflict Avoidance in Social Navigation -- a Survey](https://mlsanity.com/paper/2106.12113/) (2021)
- [CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos](https://mlsanity.com/paper/2303.09713/) (2023)
- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization](https://mlsanity.com/paper/2205.12381/) (2022)
- [NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities](https://mlsanity.com/paper/2311.01454/) (2023)
- [Interaction-Grounded Learning with Action-inclusive Feedback](https://mlsanity.com/paper/2206.08364/) (2022)
- [From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learning](https://mlsanity.com/paper/2311.10284/) (2023)
- [Assisted Robust Reward Design](https://mlsanity.com/paper/2111.09884/) (2021)
- [X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback](https://mlsanity.com/paper/2203.02072/) (2022)
- [Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces](https://mlsanity.com/paper/2211.03539/) (2022)
- [Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaboration](https://mlsanity.com/paper/2411.02599/) (2024)
- [Improving Reinforcement Learning with Human Assistance: An Argument for Human Subject Studies with HIPPO Gym](https://mlsanity.com/paper/2102.02639/) (2021)
- [Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitation](https://mlsanity.com/paper/2501.01367/) (2025)
- [Multi-trainer Interactive Reinforcement Learning System](https://mlsanity.com/paper/2210.08050/) (2022)
- [Predicting Human Perceptions of Robot Performance During Navigation Tasks](https://mlsanity.com/paper/2310.11590/) (2023)
- [How Much Progress Did I Make? An Unexplored Human Feedback Signal for Teaching Robots](https://mlsanity.com/paper/2407.06459/) (2024)
- [Robot Behavior Personalization from Sparse User Feedback](https://mlsanity.com/paper/2410.19219/) (2024)
- [A MultiModal Social Robot Toward Personalized Emotion Interaction](https://mlsanity.com/paper/2110.05186/) (2021)
- [Teach Me How to Learn: A Perspective Review towards User-centered Neuro-symbolic Learning for Robotic Surgical Systems](https://mlsanity.com/paper/2307.03853/) (2023)
- [The Social Context of Human-Robot Interactions](https://mlsanity.com/paper/2508.13982/) (2025)
- [Towards Reinforcement Learning from Neural Feedback: Mapping fNIRS Signals to Agent Performance](https://mlsanity.com/paper/2511.12844/) (2025)
- [Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systems](https://mlsanity.com/paper/2309.05787/) (2023)
- [Prosody as a Teaching Signal for Agent Learning: Exploratory Studies and Algorithmic Implications](https://mlsanity.com/paper/2410.23554/) (2024)
- [NatSGLD: A Dataset with Speech, Gesture, Logic, and Demonstration for Robot Learning in Natural Human-Robot Interaction](https://mlsanity.com/paper/2502.16718/) (2025)

## Abstract (arXiv)

Reactions such as gestures, facial expressions, and vocalizations are an abundant, naturally occurring channel of information that humans provide during interactions. A robot or other agent could leverage an understanding of such implicit human feedback to improve its task performance at no cost to the human. This approach contrasts with common agent teaching methods based on demonstrations, critiques, or other guidance that need to be attentively and intentionally provided. In this paper, we first define the general problem of learning from implicit human feedback and then propose to address this problem through a novel data-driven framework, EMPATHIC. This two-stage method consists of (1) mapping implicit human feedback to relevant task statistics such as reward, optimality, and advantage; and (2) using such a mapping to learn a task. We instantiate the first stage and three second-stage evaluations of the learned mapping. To do so, we collect a dataset of human facial reactions while participants observe an agent execute a sub-optimal policy for a prescribed training task. We train a deep neural network on this data and demonstrate its ability to (1) infer relative reward ranking of events in the training task from prerecorded human facial reactions; (2) improve the policy of an agent in the training task using live human facial reactions; and (3) transfer to a novel domain in which it evaluates robot manipulation trajectories.