mlsanityData to 7 Oct 2026

A Divergence Minimization Perspective on Imitation Learning Methods

arXiv 1911.02256 ↗First posted 6 Nov 2019cs.LG stat.MLcited by 188

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

In many settings, it is desirable to learn decision-making and control policies through learning or bootstrapping from expert demonstrations. The most common approaches under this Imitation Learning (IL) framework are Behavioural Cloning (BC), and Inverse Reinforcement Learning (IRL). Recent methods for IRL have demonstrated the capacity to learn effective policies with access to a very limited set of demonstrations, a scenario in which BC methods often fail. Unfortunately, due to multiple factors of variation, directly comparing these methods does not provide adequate intuition for understanding this difference in performance. In this work, we present a unified probabilistic perspective on IL algorithms based on divergence minimization. We present f-MAX, an f-divergence generalization of AIRL [Fu et al., 2018], a state-of-the-art IRL method. f-MAX enables us to relate prior IRL methods such as GAIL [Ho & Ermon, 2016] and AIRL [Fu et al., 2018], and understand their algorithmic properties. Through the lens of divergence minimization we tease apart the differences between BC and successful IRL approaches, and empirically evaluate these nuances on simulated high-dimensional continuous control domains. Our findings conclusively identify that IRL's state-marginal matching objective contributes most to its superior performance. Lastly, we apply our new understanding of IL methods to the problem of state-marginal matching, where we demonstrate that in simulated arm pushing environments we can teach agents a diverse range of behaviours using simply hand-specified state distributions and no reward functions or expert demonstrations. For datasets and reproducing results please refer to https://github.com/KamyarGh/rlswiss/blob/master/reproducing/fmaxpaper.md .

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 16 of its 44 references that are in the map. First posted on arXiv 6 Nov 2019.

Builds on

16 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Imitation Learning as f-Divergence Minimization30 May

Other references · 15

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Improved Training of Wasserstein GANscited by 2,223
  3. 2016OpenAI Gymcited by 1,684
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  9. 2015Generative Moment Matching Networkscited by 252
  10. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  11. 2019Efficient Exploration via State Marginal Matchingcited by 186
  12. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
3 more
  1. 2015Training generative neural networks via Maximum Mean Discrepancy optimizationcited by 149
  2. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  3. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 17; 2021: 32; 2022: 33; 2023: 48; 2024: 29; 2025: 18; 2026: 11.

188 papers.

Sort:

By area:

  1. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  2. 2020Primal Wasserstein Imitation Learningcited by 120
  3. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  5. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  6. 2021What Matters for Adversarial Imitation Learning?cited by 70
  7. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  8. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  9. 2022A Survey on Model-based Reinforcement Learningcited by 65
  10. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
  11. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
  12. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
38 more
  1. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  2. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  3. 2020Modeling Human Driving Behavior through Generative Adversarial Imitation Learningcited by 43
  4. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  5. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  6. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  7. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
  8. 2020Energy-Based Imitation Learningcited by 39
  9. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
  10. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
  11. 2021Regularized Policies are Reward Robustcited by 32
  12. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  13. 2020Imitation Learning with Sinkhorn Distancescited by 30
  14. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  15. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  16. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
  17. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  18. 2020Robust Imitation Learning from Noisy Demonstrationscited by 26
  19. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  20. 2023Learning Realistic Traffic Agents in Closed-loopcited by 24
  21. 2022You Only Live Once: Single-Life Reinforcement Learningcited by 23
  22. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimizationcited by 22
  23. 2021Continuous Control with Action Quantization from Demonstrationscited by 21
  24. 2021Learning Multi-Stage Tasks with One Demonstration via Self-Replaycited by 21
  25. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  26. 2023Coherent Soft Imitation Learningcited by 21
  27. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  28. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  29. 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learningcited by 20
  30. 2023DITTO: Offline Imitation Learning with World Modelscited by 20
  31. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  32. 2021Adversarial Option-Aware Hierarchical Imitation Learningcited by 19
  33. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  34. 2021Concave Utility Reinforcement Learning with Zero-Constraint Violationscited by 18
  35. 2022A Ranking Game for Imitation Learningcited by 18
  36. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  37. 2023DiffAIL: Diffusion Adversarial Imitation Learningcited by 18
  38. 2021Hyperparameter Selection for Imitation Learningcited by 17

50 most cited of 188 All 188 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Reparameterized Variational Divergence Minimization for Stable Imitation
  2. 2018Fast Policy Learning through Imitation and Reinforcement
  3. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstrators
  4. 2021IQ-Learn: Inverse soft-Q Learning for Imitation
  5. 2022Deconfounding Imitation Learning with Variational Inference
  6. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning
  7. 2020State-only Imitation with Transition Dynamics Mismatch
  8. 2024Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker