mlsanityData to 7 Oct 2026

Off-policy Imitation Learning from Visual Inputs

arXiv 2111.04345 ↗First posted 8 Nov 2021cs.LG cs.CVcited by 2

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Recently, various successful applications utilizing expert states in imitation learning (IL) have been witnessed. However, another IL setting -- IL from visual inputs (ILfVI), which has a greater promise to be applied in reality by utilizing online visual resources, suffers from low data-efficiency and poor performance resulted from an on-policy learning manner and high-dimensional visual inputs. We propose OPIfVI (Off-Policy Imitation from Visual Inputs), which is composed of an off-policy learning manner, data augmentation, and encoder techniques, to tackle the mentioned challenges, respectively. More specifically, to improve data-efficiency, OPIfVI conducts IL in an off-policy manner, with which sampled data can be used multiple times. In addition, we enhance the stability of OPIfVI with spectral normalization to mitigate the side-effect of off-policy training. The core factor, contributing to the poor performance of ILfVI, that we think is the agent could not extract meaningful features from visual inputs. Hence, OPIfVI employs data augmentation from computer vision to help train encoders that can better extract features from visual inputs. In addition, a specific structure of gradient backpropagation for the encoder is designed to stabilize the encoder training. At last, we demonstrate that OPIfVI is able to achieve expert-level performance and outperform existing baselines no matter visual demonstrations or visual observations are provided via extensive experiments using DeepMind Control Suite.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 57 references that are in the map. First posted on arXiv 8 Nov 2021.

Builds on

35 of 57 references.

Closest ancestors · 11

Same direction, newest first

  1. 2021Visual Adversarial Imitation Learning using Variational Models16 Jul
  2. 2021Imitation by Predicting Observations8 Jul
  3. 2020On the Guaranteed Almost Equivalence between Imitation Learning from Observation and Demonstration16 Oct
  4. 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning2 Oct
  5. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning28 Jun
  6. 2020Primal Wasserstein Imitation Learning8 Jun
  7. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  8. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement10 Oct
  9. 2018An Algorithmic Perspective on Imitation Learning16 Nov
  10. 2018Generative Adversarial Imitation from Observation17 Jul
  11. 2018Behavioral Cloning from Observation4 May

Other references · 24

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  5. 2018DeepMind Control Suitecited by 935
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  8. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  9. 2020Reinforcement Learning with Augmented Datacited by 452
  10. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
  11. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  12. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
12 more
  1. 2018DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skillscited by 153
  2. 2019Learning Data Augmentation Strategies for Object Detectioncited by 139
  3. 2020Adversarial Generation of Continuous Imagescited by 135
  4. 2018Zero-Shot Visual Imitationcited by 114
  5. 2020Visual Imitation Made Easycited by 111
  6. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  7. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  8. 2018One-Shot Hierarchical Imitation Learning of Compound Visuomotor Taskscited by 32
  9. 2018One-Shot Learning of Multi-Step Tasks from Observation via Activity Localization in Auxiliary Videocited by 16
  10. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matchingcited by 15
  11. 2019Saliency Prediction on Omnidirectional Images with Generative Adversarial Imitation Learningcited by 10
  12. 2020A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operationscited by 7

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 1.

2 papers.

  1. 2023Visual Hindsight Self-Imitation Learning for Interactive Navigationcited by 5
  2. 2025Learning from Observation: A Survey of Recent Advancescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization
  2. 2021A Pragmatic Look at Deep Imitation Learning
  3. 2023Adversarial Imitation Learning from Visual Observations using Latent Information
  4. 2021Off-Policy Imitation Learning from Observations
  5. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  6. 2024Offline Imitation Learning with Model-based Reverse Augmentation
  7. 2022Adversarial Imitation Learning from Video using a State Observer
  8. 2023Robust Visual Imitation Learning with Inverse Dynamics Representations