mlsanityData to 7 Oct 2026

Policy Contrastive Imitation Learning

arXiv 2307.02829 ↗First posted 6 Jul 2023cs.LG cs.ROcited by 3

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Adversarial imitation learning (AIL) is a popular method that has recently achieved much success. However, the performance of AIL is still unsatisfactory on the more challenging tasks. We find that one of the major reasons is due to the low quality of AIL discriminator representation. Since the AIL discriminator is trained via binary classification that does not necessarily discriminate the policy from the expert in a meaningful way, the resulting reward might not be meaningful either. We propose a new method called Policy Contrastive Imitation Learning (PCIL) to resolve this issue. PCIL learns a contrastive representation space by anchoring on different policies and generates a smooth cosine-similarity-based reward. Our proposed representation learning objective can be viewed as a stronger version of the AIL objective and provide a more meaningful comparison between the agent and the policy. From a theoretical perspective, we show the validity of our method using the apprenticeship learning framework. Furthermore, our empirical evaluation on the DeepMind Control suite demonstrates that PCIL can achieve state-of-the-art performance. Finally, qualitative results suggest that PCIL builds a smoother and more meaningful representation space for imitation learning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 46 references that are in the map. First posted on arXiv 6 Jul 2023.

Builds on

34 of 46 references.

Closest ancestors · 10

Same direction, newest first

  1. 2022An Empirical Investigation of Representation Learning for Imitation16 May
  2. 2021What Matters for Adversarial Imitation Learning?1 Jun
  3. 2020Imitation Learning with Sinkhorn Distances20 Aug
  4. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning28 Jun
  5. 2020Primal Wasserstein Imitation Learning8 Jun
  6. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  7. 2019Wasserstein Adversarial Imitation Learning19 Jun
  8. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow1 Oct
  9. 2018Generative Adversarial Imitation from Observation17 Jul
  10. 2018Behavioral Cloning from Observation4 May

Other references · 24

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2020Supervised Contrastive Learningcited by 2,803
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  7. 2017Improved Training of Wasserstein GANscited by 2,223
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2018World Modelscited by 1,444
  10. 2018DeepMind Control Suitecited by 935
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
12 more
  1. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  2. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  3. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  4. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  5. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  6. 2022Masked Visual Pre-training for Motor Controlcited by 246
  7. 2016Learning to Navigate in Complex Environmentscited by 192
  8. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  9. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  10. 2017Robust Imitation of Diverse Behaviorscited by 85
  11. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  12. 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 48

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2023Imitation Learning from Observation with Automatic Discount Schedulingcited by 10
  2. 2024Visual Imitation Learning with Calibrated Contrastive Representationcited by 0
  3. 2024MRIC: Model-Based Reinforcement-Imitation Learning with Mixture-of-Codebooks for Autonomous Driving Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms
  2. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning
  3. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
  4. 2022Auto-Encoding Adversarial Imitation Learning
  5. 2019A New Framework for Query Efficient Active Imitation Learning
  6. 2023DiffAIL: Diffusion Adversarial Imitation Learning
  7. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
  8. 2020ADAIL: Adaptive Adversarial Imitation Learning