mlsanityData to 7 Oct 2026

Visual Adversarial Imitation Learning using Variational Models

arXiv 2107.08829 ↗First posted 16 Jul 2021cs.LG cs.AI cs.ROcited by 42

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Reward function specification, which requires considerable human effort and iteration, remains a major impediment for learning behaviors through deep reinforcement learning. In contrast, providing visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We consider a setting where an agent is provided a fixed dataset of visual demonstrations illustrating how to perform a task, and must learn to solve the task using the provided demonstrations and unsupervised environment interactions. This setting presents a number of challenges including representation learning for visual observations, sample complexity due to high dimensional spaces, and learning instability due to the lack of a fixed reward or learning signal. Towards addressing these challenges, we develop a variational model-based adversarial imitation learning (V-MAIL) algorithm. The model-based approach provides a strong signal for representation learning, enables sample efficiency, and improves the stability of adversarial training by enabling on-policy learning. Through experiments involving several vision-based locomotion and manipulation tasks, we find that V-MAIL learns successful visuomotor policies in a sample-efficient manner, has better stability compared to prior work, and also achieves higher asymptotic performance. We further find that by transferring the learned models, V-MAIL can learn new tasks from visual demonstrations without any additional environment interactions. All results including videos can be found online at urlhttps://sites.google.com/view/variational-mail.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 59 references that are in the map. First posted on arXiv 16 Jul 2021.

Builds on

48 of 59 references.

Closest ancestors · 11

Same direction, newest first

  1. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage6 Jun
  2. 2021MobILE: Model-Based Imitation Learning From Observation Alone22 Feb
  3. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shift4 Feb
  4. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning28 Jun
  5. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  6. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  7. 2019Learning Belief Representations for Imitation Learning in POMDPs22 Jun
  8. 2019Imitation Learning as f-Divergence Minimization30 May
  9. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
  10. 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets6 Sep
  11. 2018Behavioral Cloning from Observation4 May

Other references · 37

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2016OpenAI Gymcited by 1,684
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2016Concrete Problems in AI Safetycited by 1,582
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  10. 2016Variational Inference: A Review for Statisticianscited by 1,466
  11. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  12. 2018DeepMind Control Suitecited by 935
25 more
  1. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  6. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  7. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  8. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  9. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  10. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  11. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  12. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  13. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  14. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 154
  15. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  16. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  17. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  18. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  19. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  20. 2019ROBEL: Robotics Benchmarks for Learning with Low-Cost Robotscited by 70
  21. 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
  22. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  23. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  24. 2016Model-based Adversarial Imitation Learningcited by 29
  25. 2019Towards Learning to Imitate from a Single Video Demonstrationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 8; 2024: 13; 2025: 5; 2026: 4.

42 papers.

Sort:

By area:

  1. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  2. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  3. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
  4. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  5. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 33
  6. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  7. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  8. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  9. 2022Planning for Sample Efficient Imitation Learningcited by 20
  10. 2023DITTO: Offline Imitation Learning with World Modelscited by 20
  11. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  12. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
30 more
  1. 2023Visual Imitation Learning with Patch Rewardscited by 16
  2. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learningcited by 13
  3. 2024Diffusion Imitation from Observationcited by 12
  4. 2024Efficient Imitation Learning with Conservative World Modelscited by 11
  5. 2023SeMAIL: Eliminating Distractors in Visual Imitation via Separated Modelscited by 10
  6. 2023Adversarial Imitation Learning from Visual Observations using Latent Informationcited by 10
  7. 2024AD3: Implicit Action is the Key for World Models to Distinguish the Diverse Visual Distractorscited by 10
  8. 2024VIEW: Visual Imitation Learning with Waypointscited by 10
  9. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  10. 2024Visually Robust Adversarial Imitation Learning from Videos with Contrastive Learningcited by 9
  11. 2024Reward-free World Models for Online Imitation Learningcited by 7
  12. 2022Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learningcited by 5
  13. 2024Adversarial Imitation Learning via Boostingcited by 5
  14. 2021Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learningcited by 3
  15. 2022Latent Policies for Adversarial Imitation Learningcited by 3
  16. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  17. 2021Off-policy Imitation Learning from Visual Inputscited by 2
  18. 2024Overcoming Knowledge Barriers: Online Imitation Learning from Visual Observation with Pretrained World Modelscited by 2
  19. 2024SENSOR: Imitate Third-Person Expert's Behaviors via Active Sensoringcited by 1
  20. 2025Feasibility-aware Imitation Learning from Observations through a Hand-mounted Demonstration Interfacecited by 1
  21. 2025Beyond Domain Randomization: Event-Inspired Perception for Visually Robust Adversarial Imitation from Videoscited by 1
  22. 2025ReLAM: Learning Anticipation Model for Rewarding Visual Robotic Manipulationcited by 1
  23. 2026Learning from Demonstrations via Capability-Aware Goal Samplingcited by 1
  24. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learningcited by 1
  25. 2024Visual Imitation Learning with Calibrated Contrastive Representationcited by 0
  26. 2024Semi-Supervised One-Shot Imitation Learningcited by 0
  27. 2025Imitation Learning of Correlated Policies in Stackelberg Gamescited by 0
  28. 2025Coupled Distributional Random Expert Distillation for World Model Online Imitation Learningcited by 0
  29. 2026Latent Wasserstein Adversarial Imitation Learningcited by 0
  30. 2026Beyond Conservatism: Recoverability-Conditioned Exploration for Model-Based Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2018Injective State-Image Mapping facilitates Visual Adversarial Imitation Learning
  2. 2022Adversarial Imitation Learning from Video using a State Observer
  3. 2021Domain-Robust Visual Imitation Learning with Mutual Information Constraints
  4. 2024Diffusion-Reward Adversarial Imitation Learning
  5. 2023Sample-efficient Adversarial Imitation Learning
  6. 2022Robust Learning from Observation with Model Misspecification
  7. 2020ADAIL: Adaptive Adversarial Imitation Learning
  8. 2018Generative Adversarial Imitation from Observation