# Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective

arXiv: https://arxiv.org/abs/2312.11214 · HTML: https://mlsanity.com/paper/2312.11214/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 18 of 18)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Error Bounds of Imitating Policies and Environments](https://mlsanity.com/paper/2010.11876/) (2020)
- [Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning](https://mlsanity.com/paper/1805.10000/) (2018)
- [Socially Compliant Navigation through Raw Depth Inputs with Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1710.02543/) (2017)
- [Sample-Efficient Imitation Learning via Generative Adversarial Nets](https://mlsanity.com/paper/1809.02064/) (2018)
- [Softmax Deep Double Deterministic Policy Gradients](https://mlsanity.com/paper/2010.09177/) (2020)
- [On Computation and Generalization of Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2001.02792/) (2020)
- [When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence](https://mlsanity.com/paper/2006.13506/) (2020)
- [Generative Inverse Deep Reinforcement Learning for Online Recommendation](https://mlsanity.com/paper/2011.02248/) (2020)
- [Characterizing the Gap Between Actor-Critic and Policy Gradient](https://mlsanity.com/paper/2106.06932/) (2021)
- [Stochastic Policy Gradient Ascent in Reproducing Kernel Hilbert Spaces](https://mlsanity.com/paper/1807.11274/) (2018)
- [Reward function shape exploration in adversarial imitation learning: an empirical study](https://mlsanity.com/paper/2104.06687/) (2021)

## Built on by (most cited first, 4 of 4)

- [LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model](https://mlsanity.com/paper/2401.02330/) (2024)
- [On Generalization and Distributional Update for Mimicking Observations with Adequate Exploration](https://mlsanity.com/paper/2501.12785/) (2025)
- [Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances](https://mlsanity.com/paper/2511.03565/) (2025)
- [AirGlove: Exploring Egocentric 3D Hand Tracking and Appearance Generalization for Sensing Gloves](https://mlsanity.com/paper/2602.05159/) (2026)

## Abstract (arXiv)

Generative Adversarial Imitation Learning (GAIL) stands as a cornerstone approach in imitation learning. This paper investigates the gradient explosion in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with stochastic policy (ST-GAIL). We begin with the observation that the training can be highly unstable for DE-GAIL at the beginning of the training phase and end up divergence. Conversely, the ST-GAIL training trajectory remains consistent, reliably converging. To shed light on these disparities, we provide an explanation from a theoretical perspective. By establishing a probabilistic lower bound for GAIL, we demonstrate that gradient explosion is an inevitable outcome for DE-GAIL due to occasionally large expert-imitator policy disparity, whereas ST-GAIL does not have the issue with it. To substantiate our assertion, we illustrate how modifications in the reward function can mitigate the gradient explosion challenge. Finally, we propose CREDO, a simple yet effective strategy that clips the reward function during the training phase, allowing the GAIL to enjoy high data efficiency and stable trainability.