mlsanityData to 7 Oct 2026

Imitation Learning via Off-Policy Distribution Matching

arXiv 1912.05032 ↗First posted 10 Dec 2019cs.LG stat.MLcited by 182

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

When performing imitation learning from expert demonstrations, distribution matching is a popular approach, in which one alternates between estimating distribution ratios and then using these ratios as rewards in a standard reinforcement learning (RL) algorithm. Traditionally, estimation of the distribution ratio requires on-policy data, which has caused previous work to either be exorbitantly data-inefficient or alter the original objective in a manner that can drastically change its optimum. In this work, we show how the original distribution ratio estimation objective may be transformed in a principled manner to yield a completely off-policy objective. In addition to the data-efficiency that this provides, we are able to show that this objective also renders the use of a separate RL optimization unnecessary.Rather, an imitation policy may be learned directly from this objective without the use of explicit rewards. We call the resulting algorithm ValueDICE and evaluate it on a suite of popular imitation learning benchmarks, finding that it can achieve state-of-the-art sample efficiency and performance.

Key facts. Cited by 182 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 10 of its 26 references that are in the map. First posted on arXiv 10 Dec 2019.

Builds on

10 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Imitation Learning as f-Divergence Minimization30 May

Other references · 9

  1. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  2. 2017Improved Training of Wasserstein GANscited by 2,223
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2016End to End Learning for Self-Driving Carscited by 1,000
  6. 2018Learning Dexterous In-Hand Manipulationcited by 643
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
  9. 2019Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Realcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 5; 2021: 25; 2022: 28; 2023: 45; 2024: 34; 2025: 26; 2026: 19.

182 papers.

Sort:

By area:

  1. 2021Implicit Behavioral Cloningcited by 442
  2. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  3. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  4. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  5. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  6. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  7. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  8. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  9. 2021Off-Policy Imitation Learning from Observationscited by 70
  10. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  11. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 60
  12. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
38 more
  1. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
  2. 2024AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policiescited by 50
  3. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  4. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  5. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  6. 2023Optimal Transport for Offline Imitation Learningcited by 42
  7. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  8. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  9. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  10. 2021RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learningcited by 31
  11. 2020Imitation Learning with Sinkhorn Distancescited by 30
  12. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  13. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  14. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
  15. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  16. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  17. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  18. 2022Proximal Point Imitation Learningcited by 25
  19. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  20. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimizationcited by 22
  21. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  22. 2023Coherent Soft Imitation Learningcited by 21
  23. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  24. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  25. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  26. 2022Planning for Sample Efficient Imitation Learningcited by 20
  27. 2023DITTO: Offline Imitation Learning with World Modelscited by 20
  28. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learningcited by 20
  29. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  30. 2022A Ranking Game for Imitation Learningcited by 18
  31. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
  32. 2023DiffAIL: Diffusion Adversarial Imitation Learningcited by 18
  33. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  34. 2020Non-Adversarial Imitation Learning and its Connections to Adversarial Methodscited by 16
  35. 2023Visual Imitation Learning with Patch Rewardscited by 16
  36. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matchingcited by 15
  37. 2022Rethinking ValueDice: Does It Really Improve Performance?cited by 15
  38. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15

50 most cited of 182 All 182 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  2. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  4. 2025Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization
  5. 2025Imitation Learning as Return Distribution Matching
  6. 2023A Strong Baseline for Batch Imitation Learning
  7. 2024Efficient Imitation Learning with Conservative World Models
  8. 2021A Critique of Strictly Batch Imitation Learning