# Adversarial Imitation via Variational Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/1809.06404 · HTML: https://mlsanity.com/paper/1809.06404/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 17 Sep 2018.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 11 of 11)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Empowerment -- an Introduction](https://mlsanity.com/paper/1310.1863/) (2013)
- [Robot Autonomy for Surgery](https://mlsanity.com/paper/1707.03080/) (2017)
- [Intrinsically motivated reinforcement learning for human-robot interaction in the real-world](https://mlsanity.com/paper/1804.05259/) (2018)
- [Show, Attend and Interact: Perceivable Human-Robot Social Interaction through Neural Attention Q-Network](https://mlsanity.com/paper/1702.08626/) (2017)

## Built on by (most cited first, 30 of 36)

- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [State Alignment-based Imitation Learning](https://mlsanity.com/paper/1911.10947/) (2019)
- [f-IRL: Inverse Reinforcement Learning via State Marginal Matching](https://mlsanity.com/paper/2011.04709/) (2020)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Intrinsic Reward Driven Imitation Learning via Generative Model](https://mlsanity.com/paper/2006.15061/) (2020)
- [Learning from Imperfect Demonstrations from Agents with Varying Dynamics](https://mlsanity.com/paper/2103.05910/) (2021)
- [SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning](https://mlsanity.com/paper/2403.09110/) (2024)
- [Imitation Learning from Video by Leveraging Proprioception](https://mlsanity.com/paper/1905.09335/) (2019)
- [CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.04782/) (2023)
- [Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2106.05068/) (2021)
- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [Inverse Contextual Bandits: Learning How Behavior Evolves over Time](https://mlsanity.com/paper/2107.06317/) (2021)
- [Learning Feasibility to Imitate Demonstrators with Different Dynamics](https://mlsanity.com/paper/2110.15142/) (2021)
- [Receding Horizon Inverse Reinforcement Learning](https://mlsanity.com/paper/2206.04477/) (2022)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Time-series Generation by Contrastive Imitation](https://mlsanity.com/paper/2311.01388/) (2023)
- [Goal-Aware Generative Adversarial Imitation Learning from Imperfect Demonstration for Robotic Cloth Manipulation](https://mlsanity.com/paper/2209.10149/) (2022)
- [Efficiently Guiding Imitation Learning Agents with Human Gaze](https://mlsanity.com/paper/2002.12500/) (2020)
- [Learning Multi-Task Transferable Rewards via Variational Inverse Reinforcement Learning](https://mlsanity.com/paper/2206.09498/) (2022)
- [Learning Reward for Physical Skills using Large Language Model](https://mlsanity.com/paper/2310.14092/) (2023)
- [Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective](https://mlsanity.com/paper/2312.00054/) (2023)
- [Hindsight Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1903.07854/) (2019)
- [Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain](https://mlsanity.com/paper/2110.11443/) (2021)
- [Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning](https://mlsanity.com/paper/2310.19831/) (2023)
- [A Generalised Inverse Reinforcement Learning Framework](https://mlsanity.com/paper/2105.11812/) (2021)
- [Towards Interpretable Deep Reinforcement Learning Models via Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.16464/) (2022)
- [Auto-Encoding Adversarial Imitation Learning](https://mlsanity.com/paper/2206.11004/) (2022)
- [IBCB: Efficient Inverse Batched Contextual Bandit for Behavioral Evolution History](https://mlsanity.com/paper/2403.16075/) (2024)
- [SuPLE: Robot Learning with Lyapunov Rewards](https://mlsanity.com/paper/2411.13613/) (2024)

## Abstract (arXiv)

We consider a problem of learning the reward and policy from expert examples under unknown dynamics. Our proposed method builds on the framework of generative adversarial networks and introduces the empowerment-regularized maximum-entropy inverse reinforcement learning to learn near-optimal rewards and policies. Empowerment-based regularization prevents the policy from overfitting to expert demonstrations, which advantageously leads to more generalized behaviors that result in learning near-optimal rewards. Our method simultaneously learns empowerment through variational information maximization along with the reward and policy under the adversarial learning formulation. We evaluate our approach on various high-dimensional complex control tasks. We also test our learned rewards in challenging transfer learning problems where training and testing environments are made to be different from each other in terms of dynamics or structure. The results show that our proposed method not only learns near-optimal rewards and policies that are matching expert behavior but also performs significantly better than state-of-the-art inverse reinforcement learning algorithms.