# Diffusion-Reward Adversarial Imitation Learning

arXiv: https://arxiv.org/abs/2405.16194 · HTML: https://mlsanity.com/paper/2405.16194/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 40 of its 60 references that are in the map.
- First posted on arXiv 25 May 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Your Diffusion Model is Secretly a Zero-Shot Classifier](https://mlsanity.com/paper/2303.16203/) (2023)
- [Imitating Human Behaviour with Diffusion Models](https://mlsanity.com/paper/2301.10677/) (2023)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Goal-Conditioned Imitation Learning using Score-based Diffusion Policies](https://mlsanity.com/paper/2304.02532/) (2023)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)

## Built on by (most cited first, 20 of 20)

- [The Ingredients for Robotic Diffusion Transformers](https://mlsanity.com/paper/2410.10088/) (2024)
- [Diffusing States and Matching Scores: A New Framework for Imitation Learning](https://mlsanity.com/paper/2410.13855/) (2024)
- [Diffusion Imitation from Observation](https://mlsanity.com/paper/2410.05429/) (2024)
- [NIL: No-data Imitation Learning by Leveraging Pre-trained Video Diffusion Models](https://mlsanity.com/paper/2503.10626/) (2025)
- [MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrations](https://mlsanity.com/paper/2505.18595/) (2025)
- [Latent Conditioned Loco-Manipulation Using Motion Priors](https://mlsanity.com/paper/2509.16061/) (2025)
- [SAD-Flower: Flow Matching for Safe, Admissible, and Dynamically Consistent Planning](https://mlsanity.com/paper/2511.05355/) (2025)
- [Action-Constrained Imitation Learning](https://mlsanity.com/paper/2508.14379/) (2025)
- [RILe: Reinforced Imitation Learning](https://mlsanity.com/paper/2406.08472/) (2024)
- [On Discovering Algorithms for Adversarial Imitation Learning](https://mlsanity.com/paper/2510.00922/) (2025)
- [Restoring Noisy Demonstration for Imitation Learning With Diffusion Models](https://mlsanity.com/paper/2510.14467/) (2025)
- [Constraint-Aware Diffusion Priors for High-Fidelity and Versatile Quadruped Locomotion](https://mlsanity.com/paper/2605.08804/) (2026)
- [Diffusion Classifier-Driven Reward for Offline Preference-based Reinforcement Learning](https://mlsanity.com/paper/2503.01143/) (2025)
- [CooT: Learning to Coordinate In-Context with Coordination Transformers](https://mlsanity.com/paper/2506.23549/) (2025)
- [SD2AIL: Adversarial Imitation Learning from Synthetic Demonstrations via Diffusion Models](https://mlsanity.com/paper/2512.18583/) (2025)
- [TacSIm: A Dataset and Benchmark for Football Tactical Style Imitation](https://mlsanity.com/paper/2603.25199/) (2026)
- [Recovering Hidden Reward in Diffusion-Based Policies](https://mlsanity.com/paper/2605.00623/) (2026)
- [Adversarial Dual On-Policy Distillation from Expressive Teacher](https://mlsanity.com/paper/2605.27095/) (2026)
- [Language-Critique Imitation Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2607.01225/) (2026)
- [Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching](https://mlsanity.com/paper/2609.38955/) (2026)

## Abstract (arXiv)

Imitation learning aims to learn a policy from observing expert demonstrations without access to reward signals from environments. Generative adversarial imitation learning (GAIL) formulates imitation learning as adversarial learning, employing a generator policy learning to imitate expert behaviors and discriminator learning to distinguish the expert demonstrations from agent trajectories. Despite its encouraging results, GAIL training is often brittle and unstable. Inspired by the recent dominance of diffusion models in generative modeling, we propose Diffusion-Reward Adversarial Imitation Learning (DRAIL), which integrates a diffusion model into GAIL, aiming to yield more robust and smoother rewards for policy learning. Specifically, we propose a diffusion discriminative classifier to construct an enhanced discriminator, and design diffusion rewards based on the classifier's output for policy learning. Extensive experiments are conducted in navigation, manipulation, and locomotion, verifying DRAIL's effectiveness compared to prior imitation learning methods. Moreover, additional experimental results demonstrate the generalizability and data efficiency of DRAIL. Visualized learned reward functions of GAIL and DRAIL suggest that DRAIL can produce more robust and smoother rewards. Project page: https://nturobotlearninglab.github.io/DRAIL/