mlsanityData to 7 Oct 2026

GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedback

arXiv 2104.06600 ↗First posted 14 Apr 2021cs.LG cs.AI cs.ROcited by 2

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Deep reinforcement learning (DRL) has achieved great successes in many simulated tasks. The sample inefficiency problem makes applying traditional DRL methods to real-world robots a great challenge. Generative Adversarial Imitation Learning (GAIL) -- a general model-free imitation learning method, allows robots to directly learn policies from expert trajectories in large environments. However, GAIL shares the limitation of other imitation learning methods that they can seldom surpass the performance of demonstrations. In this paper, to address the limit of GAIL, we propose GAN-Based Interactive Reinforcement Learning (GAIRL) from demonstration and human evaluative feedback by combining the advantages of GAIL and interactive reinforcement learning. We tested our proposed method in six physics-based control tasks, ranging from simple low-dimensional control tasks -- Cart Pole and Mountain Car, to difficult high-dimensional tasks -- Inverted Double Pendulum, Lunar Lander, Hopper and HalfCheetah. Our results suggest that with both optimal and suboptimal demonstrations, a GAIRL agent can always learn a more stable policy with optimal or close to optimal performance, while the performance of the GAIL agent is upper bounded by the performance of demonstrations or even worse than it. In addition, our results indicate the reason that GAIRL is superior over GAIL is the complementary effect of demonstrations and human evaluative feedback.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 67 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

27 of 67 references.

Closest ancestors · 4

Same direction, newest first

  1. 2019Imitation Learning from Imperfect Demonstration27 Jan
  2. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow1 Oct
  3. 2018Generative Adversarial Imitation from Observation17 Jul
  4. 2018Reinforcement Learning from Imperfect Demonstrations14 Feb

Other references · 23

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016OpenAI Gymcited by 1,684
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  7. 2018Learning Dexterous In-Hand Manipulationcited by 643
  8. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  9. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  10. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  11. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  12. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 159
11 more
  1. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  2. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  3. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  4. 2017Third-Person Imitation Learningcited by 103
  5. 2017Robust Imitation of Diverse Behaviorscited by 85
  6. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  7. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  8. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  9. 2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
  10. 2019Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidancecited by 25
  11. 2020Human Trust-based Feedback Control: Dynamically varying automation transparency to optimize human-machine interactionscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 1.

2 papers.

  1. 2024CREW: Facilitating Human-AI Teaming Researchcited by 11
  2. 2022Sensor Control for Information Gain in Dynamic, Sparse and Partially Observed Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory
  2. 2021A Pragmatic Look at Deep Imitation Learning
  3. 2020On Computation and Generalization of Generative Adversarial Imitation Learning
  4. 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets
  5. 2021A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Character Control
  6. 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence
  7. 2025Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization
  8. 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning