mlsanityData to 7 Oct 2026

Imitation Learning as f-Divergence Minimization

arXiv 1905.12888 ↗First posted 30 May 2019cs.LG cs.IT cs.ROcited by 121

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We address the problem of imitation learning with multi-modal demonstrations. Instead of attempting to learn all modes, we argue that in many tasks it is sufficient to imitate any one of them. We show that the state-of-the-art methods such as GAIL and behavior cloning, due to their choice of loss function, often incorrectly interpolate between such modes. Our key insight is to minimize the right divergence between the learner and the expert state-action distributions, namely the reverse KL divergence or I-projection. We propose a general imitation learning framework for estimating and minimizing any f-Divergence. By plugging in different divergences, we are able to recover existing algorithms such as Behavior Cloning (Kullback-Leibler), GAIL (Jensen Shannon) and Dagger (Total Variation). Empirical results show that our approximate I-projection technique is able to imitate multi-modal behaviors more reliably than GAIL and behavior cloning.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 61 references that are in the map. First posted on arXiv 30 May 2019.

Builds on

28 of 61 references.

Closest ancestors · 8

Same direction, newest first

  1. 2019Provably Efficient Imitation Learning from Observation Alone27 May
  2. 2018An Algorithmic Perspective on Imitation Learning16 Nov
  3. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow1 Oct
  4. 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets6 Sep
  5. 2018Generative Adversarial Imitation from Observation17 Jul
  6. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning29 May
  7. 2018Fast Policy Learning through Imitation and Reinforcement26 May
  8. 2018Behavioral Cloning from Observation4 May

Other references · 20

  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  3. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  4. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  6. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  7. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  8. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  9. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  10. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  11. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  12. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
8 more
  1. 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
  2. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  3. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 36
  4. 2018Adversarial Imitation via Variational Inverse Reinforcement Learningcited by 36
  5. 2018SFV: Reinforcement Learning of Physical Skills from Videoscited by 20
  6. 2016Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrationscited by 17
  7. 2017f-Divergence constrained policy improvementcited by 15
  8. 2018Maximum Causal Tsallis Entropy Imitation Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 3; 2020: 12; 2021: 15; 2022: 19; 2023: 27; 2024: 18; 2025: 18; 2026: 9.

121 papers.

Sort:

By area:

  1. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  2. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  3. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  4. 2020Primal Wasserstein Imitation Learningcited by 120
  5. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  6. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  7. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  8. 2019State Alignment-based Imitation Learningcited by 73
  9. 2023Data Quality in Imitation Learningcited by 72
  10. 2022Adversarial Motion Priors Make Good Substitutes for Complex Reward Functionscited by 68
  11. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  12. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
38 more
  1. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  2. 2022A Survey on Model-based Reinforcement Learningcited by 65
  3. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  4. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
  5. 2021MobILE: Model-Based Imitation Learning From Observation Alonecited by 44
  6. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  7. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  8. 2020Energy-Based Imitation Learningcited by 39
  9. 2020Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulationcited by 37
  10. 2023Diffusion Model-Augmented Behavioral Cloningcited by 37
  11. 2025Failure Prediction at Runtime for Generative Robot Policiescited by 36
  12. 2020Imitation Learning with Sinkhorn Distancescited by 30
  13. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamicscited by 26
  14. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 26
  15. 2022Proximal Point Imitation Learningcited by 25
  16. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  17. 2023Learning Realistic Traffic Agents in Closed-loopcited by 24
  18. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  19. 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learningcited by 20
  20. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignmentcited by 19
  21. 2021AMP: Adversarial Motion Priors for Stylized Physics-Based Character Controlcited by 18
  22. 2022A Ranking Game for Imitation Learningcited by 18
  23. 2021Hyperparameter Selection for Imitation Learningcited by 17
  24. 2020Non-Adversarial Imitation Learning and its Connections to Adversarial Methodscited by 16
  25. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitionscited by 16
  26. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16
  27. 2022Rethinking ValueDice: Does It Really Improve Performance?cited by 15
  28. 2022Model-based Trajectory Stitching for Improved Offline Reinforcement Learningcited by 15
  29. 2021SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matchingcited by 14
  30. 2021On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learningcited by 14
  31. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  32. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  33. 2022Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysiscited by 13
  34. 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergencecited by 12
  35. 2021Provably Breaking the Quadratic Error Compounding Barrier in Imitation Learning, Optimallycited by 12
  36. 2023A Coupled Flow Approach to Imitation Learningcited by 12
  37. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  38. 2021Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximizationcited by 11

50 most cited of 121 All 121 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Reparameterized Variational Divergence Minimization for Stable Imitation
  2. 2022Diverse Imitation Learning via Self-Organizing Generative Models
  3. 2020State-only Imitation with Transition Dynamics Mismatch
  4. 2020Augmenting GAIL with BC for sample efficient imitation learning
  5. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
  6. 2022Deconfounding Imitation Learning with Variational Inference
  7. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning
  8. 2026Difference-Aware Retrieval Policies for Imitation Learning