# Energy-Based Imitation Learning

arXiv: https://arxiv.org/abs/2004.09395 · HTML: https://mlsanity.com/paper/2004.09395/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 30 of its 58 references that are in the map.
- First posted on arXiv 20 Apr 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 30 of 30)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [Generative Moment Matching Networks](https://mlsanity.com/paper/1502.02761/) (2015)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model](https://mlsanity.com/paper/1904.09770/) (2019)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [Energy-based Generative Adversarial Network](https://mlsanity.com/paper/1609.03126/) (2016)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [State Alignment-based Imitation Learning](https://mlsanity.com/paper/1911.10947/) (2019)
- [Neural Empirical Bayes](https://mlsanity.com/paper/1903.02334/) (2019)
- [Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation](https://mlsanity.com/paper/1905.06750/) (2019)
- [Deep Energy Estimator Networks](https://mlsanity.com/paper/1805.08306/) (2018)
- [SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards](https://mlsanity.com/paper/1905.11108/) (2019)
- [Adversarial Contrastive Estimation](https://mlsanity.com/paper/1805.03642/) (2018)
- [Implicit Generation and Generalization in Energy-Based Models](https://mlsanity.com/paper/1903.08689/) (2019)
- [Credal Networks under Maximum Entropy](https://mlsanity.com/paper/1301.3873/) (2013)
- [Regularizing Model-Based Planning with Energy-Based Models](https://mlsanity.com/paper/1910.05527/) (2019)

## Built on by (most cited first, 30 of 40)

- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [Goal-Conditioned Imitation Learning using Score-based Diffusion Policies](https://mlsanity.com/paper/2304.02532/) (2023)
- [Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling](https://mlsanity.com/paper/2209.14548/) (2022)
- [Policy Representation via Diffusion Probability Model for Reinforcement Learning](https://mlsanity.com/paper/2305.13122/) (2023)
- [From Play to Policy: Conditional Behavior Generation from Uncurated Robot Data](https://mlsanity.com/paper/2210.10047/) (2022)
- [f-IRL: Inverse Reinforcement Learning via State Marginal Matching](https://mlsanity.com/paper/2011.04709/) (2020)
- [Visual Whole-Body Control for Legged Loco-Manipulation](https://mlsanity.com/paper/2403.16967/) (2024)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [Vision-Language Navigation with Energy-Based Policy](https://mlsanity.com/paper/2410.14250/) (2024)
- [Joint Metrics Matter: A Better Standard for Trajectory Forecasting](https://mlsanity.com/paper/2305.06292/) (2023)
- [Visual Imitation Learning with Patch Rewards](https://mlsanity.com/paper/2302.00965/) (2023)
- [Diffusing States and Matching Scores: A New Framework for Imitation Learning](https://mlsanity.com/paper/2410.13855/) (2024)
- [Invariant Causal Imitation Learning for Generalizable Policies](https://mlsanity.com/paper/2311.01489/) (2023)
- [Generative Models in Decision Making: A Survey](https://mlsanity.com/paper/2502.17100/) (2025)
- [Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games](https://mlsanity.com/paper/2106.04958/) (2021)
- [Imitation by Predicting Observations](https://mlsanity.com/paper/2107.03851/) (2021)
- [Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysis](https://mlsanity.com/paper/2208.01899/) (2022)
- [TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specification](https://mlsanity.com/paper/2209.00853/) (2022)
- [CEIL: Generalized Contextual Imitation Learning](https://mlsanity.com/paper/2306.14534/) (2023)
- [Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization](https://mlsanity.com/paper/2203.02214/) (2022)
- [Revisiting Energy Based Models as Policies: Ranking Noise Contrastive Estimation and Interpolating Energy Models](https://mlsanity.com/paper/2309.05803/) (2023)
- [RITA: Boost Driving Simulators with Realistic Interactive Traffic Flow](https://mlsanity.com/paper/2211.03408/) (2022)
- [Imitation Learning by State-Only Distribution Matching](https://mlsanity.com/paper/2202.04332/) (2022)
- [Grounded Answers for Multi-agent Decision-making Problem through Generative World Model](https://mlsanity.com/paper/2410.02664/) (2024)
- [Good Better Best: Self-Motivated Imitation Learning for noisy Demonstrations](https://mlsanity.com/paper/2310.15815/) (2023)
- [Few-Shot Task Learning through Inverse Generative Modeling](https://mlsanity.com/paper/2411.04987/) (2024)
- [SR-Reward: Taking The Path More Traveled](https://mlsanity.com/paper/2501.02330/) (2025)

## Abstract (arXiv)

We tackle a common scenario in imitation learning (IL), where agents try to recover the optimal policy from expert demonstrations without further access to the expert or environment reward signals. Except the simple Behavior Cloning (BC) that adopts supervised learning followed by the problem of compounding error, previous solutions like inverse reinforcement learning (IRL) and recent generative adversarial methods involve a bi-level or alternating optimization for updating the reward function and the policy, suffering from high computational cost and training instability. Inspired by recent progress in energy-based model (EBM), in this paper, we propose a simplified IL framework named Energy-Based Imitation Learning (EBIL). Instead of updating the reward and policy iteratively, EBIL breaks out of the traditional IRL paradigm by a simple and flexible two-stage solution: first estimating the expert energy as the surrogate reward function through score matching, then utilizing such a reward for learning the policy by reinforcement learning algorithms. EBIL combines the idea of both EBM and occupancy measure matching, and via theoretic analysis we reveal that EBIL and Max-Entropy IRL (MaxEnt IRL) approaches are two sides of the same coin, and thus EBIL could be an alternative of adversarial IRL methods. Extensive experiments on qualitative and quantitative evaluations indicate that EBIL is able to recover meaningful and interpretative reward signals while achieving effective and comparable performance against existing algorithms on IL benchmarks.