mlsanityData to 7 Oct 2026

Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning

arXiv 2606.22056 ↗First posted 20 Jun 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Adversarial imitation learning (AIL) achieves high-quality imitation compared to behavioral cloning (BC), but demands substantial online environment interaction. Recent empirical work has explored initializing AIL algorithms with BC pretrained policies to address this limitation, yet a rigorous theoretical understanding of pretraining's role in AIL remains elusive. This paper provides a systematic theoretical analysis and introduces principled pretraining algorithms for accelerating AIL. We begin by analyzing AIL with policy pretraining alone, identifying reward error as the dominant source of suboptimality. This reveals a critical and previously overlooked gap: the absence of reward pretraining. Motivated by this finding, we develop a principled policy-reward co-pretraining approach grounded in a reward shaping analysis. Our analysis uncovers a fundamental connection between expert policies and shaping rewards, which naturally gives rise to CoPT-AIL, an approach that jointly pretrains both policy and reward through a single BC procedure. We prove that CoPT-AIL achieves an improved imitation gap bound over standard AIL, establishing the first theoretical guarantee for the benefits of pretraining in AIL. Experimental results confirm CoPT-AIL's superior performance over existing AIL methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 61 references that are in the map. First posted on arXiv 20 Jun 2026.

Builds on

39 of 61 references.

Closest ancestors · 17

Same direction, newest first

  1. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms3 May
  2. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning24 May
  3. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptions3 May
  4. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions11 Jun
  5. 2023Coherent Soft Imitation Learning25 May
  6. 2022Proximal Point Imitation Learning22 Sep
  7. 2022Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysis3 Aug
  8. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transport30 Jun
  9. 2021Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation19 Aug
  10. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  11. 2021What Matters for Adversarial Imitation Learning?1 Jun
  12. 2020Error Bounds of Imitating Policies and Environments22 Oct
5 more
  1. 2020Toward the Fundamental Limits of Imitation Learning13 Sep
  2. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  3. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  4. 2018An Algorithmic Perspective on Imitation Learning16 Nov
  5. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  6. 2018DeepMind Control Suitecited by 935
  7. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 407
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  9. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  10. 2019A Theory of Regularized Markov Decision Processescited by 279
  11. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  12. 2020Optimistic Policy Optimization with Bandit Feedbackcited by 79
10 more
  1. 2021Identifiability in inverse reinforcement learningcited by 55
  2. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 35
  3. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  4. 2024Hybrid Inverse Reinforcement Learningcited by 31
  5. 2021Online Apprenticeship Learningcited by 28
  6. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  7. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  8. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  9. 2023Demonstration-Regularized RLcited by 8
  10. 2025ReLAM: Learning Anticipation Model for Rewarding Visual Robotic Manipulationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Provable Benefits of Regularization: Fast Rates for Adversarial Imitation Learning
  2. 2022Planning for Sample Efficient Imitation Learning
  3. 2023Policy Contrastive Imitation Learning
  4. 2025On Discovering Algorithms for Adversarial Imitation Learning
  5. 2021On Generalization of Adversarial Imitation Learning and Beyond
  6. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
  7. 2019A New Framework for Query Efficient Active Imitation Learning
  8. 2022Auto-Encoding Adversarial Imitation Learning