mlsanityData to 7 Oct 2026

Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective

arXiv 2312.11214 ↗First posted 18 Dec 2023cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Generative Adversarial Imitation Learning (GAIL) stands as a cornerstone approach in imitation learning. This paper investigates the gradient explosion in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with stochastic policy (ST-GAIL). We begin with the observation that the training can be highly unstable for DE-GAIL at the beginning of the training phase and end up divergence. Conversely, the ST-GAIL training trajectory remains consistent, reliably converging. To shed light on these disparities, we provide an explanation from a theoretical perspective. By establishing a probabilistic lower bound for GAIL, we demonstrate that gradient explosion is an inevitable outcome for DE-GAIL due to occasionally large expert-imitator policy disparity, whereas ST-GAIL does not have the issue with it. To substantiate our assertion, we illustrate how modifications in the reward function can mitigate the gradient explosion challenge. Finally, we propose CREDO, a simple yet effective strategy that clips the reward function during the training phase, allowing the GAIL to enjoy high data efficiency and stable trainability.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 18 of its 35 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

18 of 35 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021Reward function shape exploration in adversarial imitation learning: an empirical study14 Apr
  2. 2020Error Bounds of Imitating Policies and Environments22 Oct
  3. 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence24 Jun
  4. 2020On Computation and Generalization of Generative Adversarial Imitation Learning9 Jan
  5. 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets6 Sep

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  9. 2017Socially Compliant Navigation through Raw Depth Inputs with Generative Adversarial Imitation Learningcited by 50
  10. 2020Softmax Deep Double Deterministic Policy Gradientscited by 31
  11. 2020Generative Inverse Deep Reinforcement Learning for Online Recommendationcited by 12
  12. 2021Characterizing the Gap Between Actor-Critic and Policy Gradientcited by 9
1 more
  1. 2018Stochastic Policy Gradient Ascent in Reproducing Kernel Hilbert Spacescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  2. 2025On Generalization and Distributional Update for Mimicking Observations with Adequate Explorationcited by 0
  3. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advancescited by 0
  4. 2026AirGlove: Exploring Egocentric 3D Hand Tracking and Appearance Generalization for Sensing Glovescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Generative Adversarial Imitation Learning with Neural Networks: Global Optimality and Convergence Rate
  2. 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning
  3. 2024C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory
  4. 2021Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
  5. 2018Risk-Sensitive Generative Adversarial Imitation Learning
  6. 2024Diffusion-Reward Adversarial Imitation Learning
  7. 2021Adversarial Option-Aware Hierarchical Imitation Learning
  8. 2021A Pragmatic Look at Deep Imitation Learning