mlsanityData to 7 Oct 2026

Energy-Based Imitation Learning

arXiv 2004.09395 ↗First posted 20 Apr 2020cs.LG stat.MLcited by 39

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We tackle a common scenario in imitation learning (IL), where agents try to recover the optimal policy from expert demonstrations without further access to the expert or environment reward signals. Except the simple Behavior Cloning (BC) that adopts supervised learning followed by the problem of compounding error, previous solutions like inverse reinforcement learning (IRL) and recent generative adversarial methods involve a bi-level or alternating optimization for updating the reward function and the policy, suffering from high computational cost and training instability. Inspired by recent progress in energy-based model (EBM), in this paper, we propose a simplified IL framework named Energy-Based Imitation Learning (EBIL). Instead of updating the reward and policy iteratively, EBIL breaks out of the traditional IRL paradigm by a simple and flexible two-stage solution: first estimating the expert energy as the surrogate reward function through score matching, then utilizing such a reward for learning the policy by reinforcement learning algorithms. EBIL combines the idea of both EBM and occupancy measure matching, and via theoretic analysis we reveal that EBIL and Max-Entropy IRL (MaxEnt IRL) approaches are two sides of the same coin, and thus EBIL could be an alternative of adversarial IRL methods. Extensive experiments on qualitative and quantitative evaluations indicate that EBIL is able to recover meaningful and interpretative reward signals while achieving effective and comparable performance against existing algorithms on IL benchmarks.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 58 references that are in the map. First posted on arXiv 20 Apr 2020.

Builds on

30 of 58 references.

Closest ancestors · 7

Same direction, newest first

  1. 2019State Alignment-based Imitation Learning21 Nov
  2. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  3. 2019Imitation Learning as f-Divergence Minimization30 May
  4. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards27 May
  5. 2019Provably Efficient Imitation Learning from Observation Alone27 May
  6. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation16 May
  7. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2017Improved Training of Wasserstein GANscited by 2,223
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  11. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  12. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 381
11 more
  1. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  2. 2015Generative Moment Matching Networkscited by 252
  3. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  4. 2019Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Modelcited by 147
  5. 2016Energy-based Generative Adversarial Networkcited by 119
  6. 2019Neural Empirical Bayescited by 46
  7. 2018Deep Energy Estimator Networkscited by 34
  8. 2018Adversarial Contrastive Estimationcited by 23
  9. 2019Implicit Generation and Generalization in Energy-Based Modelscited by 21
  10. 2013Credal Networks under Maximum Entropycited by 13
  11. 2019Regularizing Model-Based Planning with Energy-Based Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 3; 2022: 8; 2023: 11; 2024: 10; 2025: 4; 2026: 1.

39 papers.

Sort:

By area:

  1. 2021Implicit Behavioral Cloningcited by 442
  2. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  3. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  4. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  5. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  6. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  7. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  8. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  9. 2024Visual Whole-Body Control for Legged Loco-Manipulationcited by 58
  10. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
  11. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  12. 2024Vision-Language Navigation with Energy-Based Policycited by 19
27 more
  1. 2023Joint Metrics Matter: A Better Standard for Trajectory Forecastingcited by 17
  2. 2023Visual Imitation Learning with Patch Rewardscited by 16
  3. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16
  4. 2023Invariant Causal Imitation Learning for Generalizable Policiescited by 15
  5. 2021Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Gamescited by 14
  6. 2025Generative Models in Decision Making: A Surveycited by 14
  7. 2021Imitation by Predicting Observationscited by 13
  8. 2022Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysiscited by 13
  9. 2022TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specificationcited by 13
  10. 2022Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimizationcited by 12
  11. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  12. 2023Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Modelscited by 9
  13. 2022RITA: Boost Driving Simulators with Realistic Interactive Traffic Flowcited by 7
  14. 2022Imitation Learning by State-Only Distribution Matchingcited by 6
  15. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  16. 2023Good Better Best: Self-Motivated Imitation Learning for noisy Demonstrationscited by 4
  17. 2024Few-Shot Task Learning through Inverse Generative Modelingcited by 4
  18. 2025SR-Reward: Taking The Path More Traveledcited by 3
  19. 2023Bayesian Reparameterization of Reward-Conditioned Reinforcement Learning with Energy-based Modelscited by 2
  20. 2024Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learningcited by 2
  21. 2024Rethinking the Intermediate Features in Adversarial Attacks: Misleading Robotic Models via Adversarial Distillationcited by 1
  22. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignmentcited by 1
  23. 2023A Novel Variational Lower Bound for Inverse Reinforcement Learningcited by 0
  24. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proofcited by 0
  25. 2024On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theorycited by 0
  26. 2025Beyond the Proxy: Trajectory-Distilled Guidance for Offline GFlowNet Trainingcited by 0
  27. 2025Planning as Descent: Goal-Conditioned Latent Trajectory Synthesis in Learned Energy Landscapescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Noise-conditioned Energy-based Annealed Rewards (NEAR): A Generative Framework for Imitation Learning from Observation
  2. 2021Reinforced Imitation Learning by Free Energy Principle
  3. 2026Recovering Hidden Reward in Diffusion-Based Policies
  4. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
  5. 2021A Critique of Strictly Batch Imitation Learning
  6. 2019A New Framework for Query Efficient Active Imitation Learning
  7. 2022A Ranking Game for Imitation Learning
  8. 2024Hybrid Inverse Reinforcement Learning