mlsanityData to 14 Oct 2026

Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning

arXiv 1809.02925 ↗First posted 9 Sep 2018cs.LG stat.MLcited by 204

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We identify two issues with the family of algorithms based on the Adversarial Imitation Learning framework. The first problem is implicit bias present in the reward functions used in these algorithms. While these biases might work well for some environments, they can also lead to sub-optimal behavior in others. Secondly, even though these algorithms can learn from few expert demonstrations, they require a prohibitively large number of interactions with the environment in order to imitate the expert for many real-world applications. In order to address these issues, we propose a new algorithm called Discriminator-Actor-Critic that uses off-policy Reinforcement Learning to reduce policy-environment interaction sample complexity by an average factor of 10. Furthermore, since our reward function is designed to be unbiased, we can apply our algorithm to many problems without making any task-specific adjustments.

Key facts. Cited by 204 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 40 references that are in the map. First posted on arXiv 9 Sep 2018.

Builds on

21 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Generative Adversarial Imitation from Observation17 Jul

Other references · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2017Improved Training of Wasserstein GANscited by 2,235
  7. 2015Variational Inference with Normalizing Flowscited by 2,170
  8. 2016Generative Adversarial Imitation Learningcited by 1,597
  9. 2017Deep Reinforcement Learning that Matterscited by 754
  10. 2017Wasserstein GANcited by 639
  11. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  12. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
8 more
  1. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  2. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  3. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  4. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
  5. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  6. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  7. 2017Robust Imitation of Diverse Behaviorscited by 86
  8. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 10; 2020: 16; 2021: 39; 2022: 35; 2023: 38; 2024: 35; 2025: 17; 2026: 14.

204 papers.

Sort:

By area:

  1. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  2. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  3. 2019Goal-conditioned Imitation Learningcited by 162
  4. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 122
  5. 2020Primal Wasserstein Imitation Learningcited by 121
  6. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  7. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  8. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  9. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 76
  10. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 75
  11. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  12. 2021Off-Policy Imitation Learning from Observationscited by 70
38 more
  1. 2021What Matters for Adversarial Imitation Learning?cited by 70
  2. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
  3. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 67
  4. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  5. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
  6. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 49
  7. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  8. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 47
  9. 2021Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularizationcited by 46
  10. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  11. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  12. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  13. 2023Optimal Transport for Offline Imitation Learningcited by 42
  14. 2020Energy-Based Imitation Learningcited by 40
  15. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  16. 2023Diffusion Model-Augmented Behavioral Cloningcited by 38
  17. 2019Positive-Unlabeled Reward Learningcited by 37
  18. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 35
  19. 2022imitation: Clean Imitation Learning Implementationscited by 34
  20. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 34
  21. 2021RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learningcited by 31
  22. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 31
  23. 2021Online Apprenticeship Learningcited by 29
  24. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 29
  25. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewardscited by 28
  26. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  27. 2022Robust Imitation Learning against Variations in Environment Dynamicscited by 26
  28. 2022Proximal Point Imitation Learningcited by 26
  29. 2022Scalable Multi-Agent Model-Based Reinforcement Learningcited by 25
  30. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  31. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  32. 2019Imitation Learning from Video by Leveraging Proprioceptioncited by 23
  33. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimizationcited by 22
  34. 2021Continuous Control with Action Quantization from Demonstrationscited by 22
  35. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 22
  36. 2023For Pre-Trained Vision Models in Motor Control, Not All Policy Learning Methods are Created Equalcited by 22
  37. 2023Coherent Soft Imitation Learningcited by 22
  38. 2021A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directionscited by 21

50 most cited of 204 All 204 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization
  2. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations
  3. 2023Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning
  4. 2021Imitation Learning by Reinforcement Learning
  5. 2025On Discovering Algorithms for Adversarial Imitation Learning
  6. 2019A New Framework for Query Efficient Active Imitation Learning
  7. 2023Sample-efficient Adversarial Imitation Learning
  8. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation