mlsanityData to 7 Oct 2026

Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guarantees

arXiv 2405.16668 ↗First posted 26 May 2024cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Adversarial Imitation Learning (AIL) faces challenges with sample inefficiency because of its reliance on sufficient on-policy data to evaluate the performance of the current policy during reward function updates. In this work, we study the convergence properties and sample complexity of off-policy AIL algorithms. We show that, even in the absence of importance sampling correction, reusing samples generated by the o(√K) most recent policies, where K is the number of iterations of policy updates and reward updates, does not undermine the convergence guarantees of this class of algorithms. Furthermore, our results indicate that the distribution shift error induced by off-policy updates is dominated by the benefits of having more data available. This result provides theoretical support for the sample efficiency of off-policy AIL algorithms. To the best of our knowledge, this is the first work that provides theoretical guarantees for off-policy AIL algorithms.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 36 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

29 of 36 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Adversarial Imitation Learning from Visual Observations using Latent Information29 Sep
  2. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions11 Jun
  3. 2023Visual Imitation Learning with Patch Rewards2 Feb
  4. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching4 Feb
  5. 2021Off-Policy Imitation Learning from Observations25 Feb
  6. 2020Error Bounds of Imitating Policies and Environments22 Oct
  7. 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence24 Jun
  8. 2020On Computation and Generalization of Generative Adversarial Imitation Learning9 Jan
  9. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  10. 2019On the Global Convergence of Imitation Learning: A Case for Linear Quadratic Regulator11 Jan
  11. 2018An Algorithmic Perspective on Imitation Learning16 Nov
  12. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2017Constrained Policy Optimizationcited by 928
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  5. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  7. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  8. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  9. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  10. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  11. 2020Mirror Descent Policy Optimizationcited by 102
  12. 2021The best of both worlds: stochastic and adversarial episodic MDPs with unknown transitioncited by 40
5 more
  1. 2021Online Apprenticeship Learningcited by 28
  2. 2021Near-optimal Policy Optimization Algorithms for Learning Adversarial Linear Mixture MDPscited by 28
  3. 2016Learning Transferable Policies for Monocular Reactive MAV Controlcited by 18
  4. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matchingcited by 15
  5. 2020Refined Analysis of FPL for Adversarial Markov Decision Processescited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learningcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization
  2. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms
  3. 2020Off-Policy Adversarial Inverse Reinforcement Learning
  4. 2021Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency
  5. 2021On Generalization of Adversarial Imitation Learning and Beyond
  6. 2024Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximation
  7. 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets
  8. 2022Planning for Sample Efficient Imitation Learning