mlsanityData to 7 Oct 2026

Strictly Batch Imitation Learning by Energy-based Distribution Matching

arXiv 2006.14154 ↗First posted 25 Jun 2020stat.ML cs.LGcited by 54

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Consider learning a policy purely on the basis of demonstrated behavior -- that is, with no access to reinforcement signals, no knowledge of transition dynamics, and no further interaction with the environment. This *strictly batch imitation learning* problem arises wherever live experimentation is costly, such as in healthcare. One solution is simply to retrofit existing algorithms for apprenticeship learning to work in the offline setting. But such an approach leans heavily on off-policy evaluation or offline model estimation, and can be indirect and inefficient. We argue that a good solution should be able to explicitly parameterize a policy (i.e. respecting action conditionals), implicitly learn from rollout dynamics (i.e. leveraging state marginals), and -- crucially -- operate in an entirely offline fashion. To address this challenge, we propose a novel technique by *energy-based distribution matching* (EDM): By identifying parameterizations of the (discriminative) model of a policy with the (generative) energy function for state distributions, EDM yields a simple but effective solution that equivalently minimizes a divergence between the occupancy measure for the demonstrator and a model thereof for the imitator. Through experiments with application to control and healthcare settings, we illustrate consistent performance gains over existing algorithms for strictly batch imitation learning.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 73 references that are in the map. First posted on arXiv 25 Jun 2020.

Builds on

29 of 73 references.

Closest ancestors · 11

Same direction, newest first

  1. 2020Primal Wasserstein Imitation Learning8 Jun
  2. 2020Energy-Based Imitation Learning20 Apr
  3. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  4. 2019State Alignment-based Imitation Learning21 Nov
  5. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  6. 2019Wasserstein Adversarial Imitation Learning19 Jun
  7. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards27 May
  8. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation16 May
  9. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
  10. 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets6 Sep
  11. 2018Global overview of Imitation Learning19 Jan

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016OpenAI Gymcited by 1,684
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  6. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  7. 2016A Theory of Generative ConvNetcited by 174
  8. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  9. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  10. 2018Adversarial Imitation via Variational Inverse Reinforcement Learningcited by 36
  11. 2016Model-based Adversarial Imitation Learningcited by 29
  12. 2019Truly Batch Apprenticeship Learning with Deep Successor Featurescited by 27
6 more
  1. 2016Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamicscited by 24
  2. 2020How to Train Your Energy-Based Model for Regressioncited by 23
  3. 2019Implicit Generation and Generalization in Energy-Based Modelscited by 21
  4. 2016Smooth Imitation Learning for Online Sequence Predictioncited by 9
  5. 2016Density Matching Reward Learningcited by 4
  6. 2013Probabilistic inverse reinforcement learning in unknown environmentscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 7; 2023: 13; 2024: 16; 2025: 5; 2026: 3.

54 papers.

Sort:

By area:

  1. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  2. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  3. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  4. 2022Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulationcited by 61
  5. 2024Equivariant Diffusion Policycited by 61
  6. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  7. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  8. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  9. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  10. 2024CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Predictioncited by 26
  11. 2022Proximal Point Imitation Learningcited by 25
  12. 2021Continuous Control with Action Quantization from Demonstrationscited by 21
38 more
  1. 2024Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusioncited by 21
  2. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learningcited by 20
  3. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyondcited by 20
  4. 2024Vision-Language Navigation with Energy-Based Policycited by 19
  5. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  6. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  7. 2023Invariant Causal Imitation Learning for Generalizable Policiescited by 15
  8. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  9. 2021Imitation by Predicting Observationscited by 13
  10. 2021Inverse Contextual Bandits: Learning How Behavior Evolves over Timecited by 12
  11. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  12. 2024Supervised Fine-Tuning as Inverse Reinforcement Learningcited by 12
  13. 2021The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulationcited by 11
  14. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learningcited by 11
  15. 2023Inverse Decision Modeling: Learning Interpretable Representations of Behaviorcited by 10
  16. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
  17. 2025Robotic Compliant Object Prying Using Diffusion Policy Guided by Vision and Force Observationscited by 9
  18. 2024SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulationcited by 8
  19. 2023C3DM: Constrained-Context Conditional Diffusion Models for Imitation Learningcited by 7
  20. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  21. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
  22. 2023Curricular Subgoals for Inverse Reinforcement Learningcited by 5
  23. 2024LeTO: Learning Constrained Visuomotor Policy with Differentiable Trajectory Optimizationcited by 5
  24. 2021Imitation Learning with Human Eye Gaze via Multi-Objective Predictioncited by 4
  25. 2021A Critique of Strictly Batch Imitation Learningcited by 4
  26. 2022Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experiencecited by 4
  27. 2024Walking the Values in Bayesian Inverse Reinforcement Learningcited by 4
  28. 2023Explaining by Imitating: Understanding Decisions by Interpretable Policy Learningcited by 3
  29. 2024OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequencescited by 3
  30. 2024Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learningcited by 3
  31. 2023Conditional Kernel Imitation Learning for Continuous State Environmentscited by 2
  32. 2024A Generalized Apprenticeship Learning Framework for Modeling Heterogeneous Student Pedagogical Strategiescited by 2
  33. 2025Language Model Distillation: A Temporal Difference Imitation Learning Perspectivecited by 2
  34. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learningcited by 2
  35. 2026A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategiescited by 2
  36. 2023Replicating Complex Dialogue Policy of Humans via Offline Imitation Learning with Supervised Regularizationcited by 1
  37. 2023An Offline Time-aware Apprenticeship Learning Framework for Evolving Reward Functionscited by 1
  38. 2025Learning dissection trajectories from expert surgical videos via imitation learning with equivariant diffusioncited by 1

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  2. 2023A Strong Baseline for Batch Imitation Learning
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  4. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  5. 2024Augmenting Offline Reinforcement Learning with State-only Interactions
  6. 2024Offline Imitation Learning with Model-based Reverse Augmentation
  7. 2024Efficient Imitation Learning with Conservative World Models
  8. 2023Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models