# GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedback

arXiv: https://arxiv.org/abs/2104.06600 · HTML: https://mlsanity.com/paper/2104.06600/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 27 of its 67 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 27 of 27)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Stochastic Neural Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1704.03012/) (2017)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow](https://mlsanity.com/paper/1810.00821/) (2018)
- [Third-Person Imitation Learning](https://mlsanity.com/paper/1703.01703/) (2017)
- [Robust Imitation of Diverse Behaviors](https://mlsanity.com/paper/1707.02747/) (2017)
- [Reinforcement Learning from Imperfect Demonstrations](https://mlsanity.com/paper/1802.05313/) (2018)
- [Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences](https://mlsanity.com/paper/2006.14091/) (2020)
- [Deep Reinforcement Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1902.04257/) (2019)
- [Model-Free Imitation Learning with Policy Optimization](https://mlsanity.com/paper/1605.08478/) (2016)
- [When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans](https://mlsanity.com/paper/2001.04377/) (2020)
- [Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance](https://mlsanity.com/paper/1911.07109/) (2019)
- [Human Trust-based Feedback Control: Dynamically varying automation transparency to optimize human-machine interactions](https://mlsanity.com/paper/2006.16353/) (2020)

## Built on by (most cited first, 2 of 2)

- [CREW: Facilitating Human-AI Teaming Research](https://mlsanity.com/paper/2408.00170/) (2024)
- [Sensor Control for Information Gain in Dynamic, Sparse and Partially Observed Environments](https://mlsanity.com/paper/2211.01527/) (2022)

## Abstract (arXiv)

Deep reinforcement learning (DRL) has achieved great successes in many simulated tasks. The sample inefficiency problem makes applying traditional DRL methods to real-world robots a great challenge. Generative Adversarial Imitation Learning (GAIL) -- a general model-free imitation learning method, allows robots to directly learn policies from expert trajectories in large environments. However, GAIL shares the limitation of other imitation learning methods that they can seldom surpass the performance of demonstrations. In this paper, to address the limit of GAIL, we propose GAN-Based Interactive Reinforcement Learning (GAIRL) from demonstration and human evaluative feedback by combining the advantages of GAIL and interactive reinforcement learning. We tested our proposed method in six physics-based control tasks, ranging from simple low-dimensional control tasks -- Cart Pole and Mountain Car, to difficult high-dimensional tasks -- Inverted Double Pendulum, Lunar Lander, Hopper and HalfCheetah. Our results suggest that with both optimal and suboptimal demonstrations, a GAIRL agent can always learn a more stable policy with optimal or close to optimal performance, while the performance of the GAIL agent is upper bounded by the performance of demonstrations or even worse than it. In addition, our results indicate the reason that GAIRL is superior over GAIL is the complementary effect of demonstrations and human evaluative feedback.