# Non-Adversarial Imitation Learning and its Connections to Adversarial Methods

arXiv: https://arxiv.org/abs/2008.03525 · HTML: https://mlsanity.com/paper/2008.03525/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 19 of its 53 references that are in the map.
- First posted on arXiv 8 Aug 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 19 of 19)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections](https://mlsanity.com/paper/1906.04733/) (2019)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [Hierarchical Variational Models](https://mlsanity.com/paper/1511.02386/) (2015)
- [Auxiliary Deep Generative Models](https://mlsanity.com/paper/1602.05473/) (2016)
- [Wasserstein Adversarial Imitation Learning](https://mlsanity.com/paper/1906.08113/) (2019)
- [The Variational Gaussian Process](https://mlsanity.com/paper/1511.06499/) (2015)
- [Expected Information Maximization: Using the I-Projection for Mixture Density Estimation](https://mlsanity.com/paper/2001.08682/) (2020)

## Built on by (most cited first, 17 of 17)

- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap](https://mlsanity.com/paper/2103.03236/) (2021)
- [Imitation Learning: Progress, Taxonomies and Challenges](https://mlsanity.com/paper/2106.12177/) (2021)
- [LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning](https://mlsanity.com/paper/2303.00599/) (2023)
- [Coherent Soft Imitation Learning](https://mlsanity.com/paper/2305.16498/) (2023)
- [A Ranking Game for Imitation Learning](https://mlsanity.com/paper/2202.03481/) (2022)
- [Imitation by Predicting Observations](https://mlsanity.com/paper/2107.03851/) (2021)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Time-series Generation by Contrastive Imitation](https://mlsanity.com/paper/2311.01388/) (2023)
- [IDIL: Imitation Learning of Intent-Driven Expert Behavior](https://mlsanity.com/paper/2404.16989/) (2024)
- [On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning](https://mlsanity.com/paper/2106.08199/) (2021)
- [Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency](https://mlsanity.com/paper/2112.06054/) (2021)
- [Versatile Inverse Reinforcement Learning via Cumulative Rewards](https://mlsanity.com/paper/2111.07667/) (2021)
- [UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations](https://mlsanity.com/paper/2410.08307/) (2024)
- [Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations](https://mlsanity.com/paper/2505.21182/) (2025)
- [Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates](https://mlsanity.com/paper/2605.11020/) (2026)
- [A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learning](https://mlsanity.com/paper/2610.14041/) (2026)

## Abstract (arXiv)

Many modern methods for imitation learning and inverse reinforcement learning, such as GAIL or AIRL, are based on an adversarial formulation. These methods apply GANs to match the expert's distribution over states and actions with the implicit state-action distribution induced by the agent's policy. However, by framing imitation learning as a saddle point problem, adversarial methods can suffer from unstable optimization, and convergence can only be shown for small policy updates. We address these problems by proposing a framework for non-adversarial imitation learning. The resulting algorithms are similar to their adversarial counterparts and, thus, provide insights for adversarial imitation learning methods. Most notably, we show that AIRL is an instance of our non-adversarial formulation, which enables us to greatly simplify its derivations and obtain stronger convergence guarantees. We also show that our non-adversarial formulation can be used to derive novel algorithms by presenting a method for offline imitation learning that is inspired by the recent ValueDice algorithm, but does not rely on small policy updates for convergence. In our simulated robot experiments, our offline method for non-adversarial imitation learning seems to perform best when using many updates for policy and discriminator at each iteration and outperforms behavioral cloning and ValueDice.