# Efficiently Guiding Imitation Learning Agents with Human Gaze

arXiv: https://arxiv.org/abs/2002.12500 · HTML: https://mlsanity.com/paper/2002.12500/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 28 Feb 2020.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Playing hard exploration games by watching YouTube](https://mlsanity.com/paper/1805.11592/) (2018)
- [Label-Free Supervision of Neural Networks with Physics and Domain Knowledge](https://mlsanity.com/paper/1609.05566/) (2016)
- [Leveraging Human Guidance for Deep Reinforcement Learning Tasks](https://mlsanity.com/paper/1909.09906/) (2019)
- [A survey of robot learning from demonstrations for Human-Robot Collaboration](https://mlsanity.com/paper/1710.08789/) (2017)
- [Adversarial Imitation via Variational Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.06404/) (2018)
- [AGIL: Learning Attention from Human for Visuomotor Tasks](https://mlsanity.com/paper/1806.03960/) (2018)
- [Inverse reinforcement learning for video games](https://mlsanity.com/paper/1810.10593/) (2018)
- [Periphery-Fovea Multi-Resolution Driving Model guided by Human Attention](https://mlsanity.com/paper/1903.09950/) (2019)
- [Learning Transferable Policies for Monocular Reactive MAV Control](https://mlsanity.com/paper/1608.00627/) (2016)
- [Towards Dropout Training for Convolutional Neural Networks](https://mlsanity.com/paper/1512.00242/) (2015)
- [Understanding Teacher Gaze Patterns for Robot Learning](https://mlsanity.com/paper/1907.07202/) (2019)
- [Gaze Training by Modulated Dropout Improves Imitation Learning](https://mlsanity.com/paper/1904.08377/) (2019)

## Built on by (most cited first, 6 of 6)

- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [Selective Eye-gaze Augmentation To Enhance Imitation Learning In Atari Games](https://mlsanity.com/paper/2012.03145/) (2020)
- [Learning Memory Mechanisms for Decision Making through Demonstrations](https://mlsanity.com/paper/2411.07954/) (2024)
- [Imitation Learning with Human Eye Gaze via Multi-Objective Prediction](https://mlsanity.com/paper/2102.13008/) (2021)
- [Improving Reinforcement Learning with Human Assistance: An Argument for Human Subject Studies with HIPPO Gym](https://mlsanity.com/paper/2102.02639/) (2021)
- [Tell me why: Training preferences-based RL with human preferences and step-level explanations](https://mlsanity.com/paper/2405.14244/) (2024)

## Abstract (arXiv)

Human gaze is known to be an intention-revealing signal in human demonstrations of tasks. In this work, we use gaze cues from human demonstrators to enhance the performance of agents trained via three popular imitation learning methods -- behavioral cloning (BC), behavioral cloning from observation (BCO), and Trajectory-ranked Reward EXtrapolation (T-REX). Based on similarities between the attention of reinforcement learning agents and human gaze, we propose a novel approach for utilizing gaze data in a computationally efficient manner, as part of an auxiliary loss function, which guides a network to have higher activations in image regions where the human's gaze fixated. This work is a step towards augmenting any existing convolutional imitation learning agent's training with auxiliary gaze data. Our auxiliary coverage-based gaze loss (CGL) guides learning toward a better reward function or policy, without adding any additional learnable parameters and without requiring gaze data at test time. We find that our proposed approach improves the performance by 95% for BC, 343% for BCO, and 390% for T-REX, averaged over 20 different Atari games. We also find that compared to a prior state-of-the-art imitation learning method assisted by human gaze (AGIL), our method achieves better performance, and is more efficient in terms of learning with fewer demonstrations. We further interpret trained CGL agents with a saliency map visualization method to explain their performance. At last, we show that CGL can help alleviate a well-known causal confusion problem in imitation learning.