VILD: Variational Imitation Learning with Diverse-quality Demonstrations
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs. IL in such situations can be challenging, especially when the level of demonstrators' expertise is unknown. We propose a new IL method called underlinevariational underlineimitation underlinelearning with underlinediverse-quality demonstrations (VILD), where we explicitly model the level of demonstrators' expertise with a probabilistic graphical model and estimate it along with a reward function. We show that a naive approach to estimation is not suitable to large state and action spaces, and fix its issues by using a variational approach which can be easily implemented using existing reinforcement learning methods. Experiments on continuous-control benchmarks demonstrate that VILD outperforms state-of-the-art methods. Our work enables scalable and data-efficient IL under more realistic settings than before.
Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 65 references that are in the map. First posted on arXiv 15 Sep 2019.
Builds on
30 of 65 references.
Closest ancestors · 5
Same direction, newest first
- 2019Generative predecessor models for sample-efficient imitation learning1 Apr
- 2019Imitation Learning from Imperfect Demonstration27 Jan
- 2018An Algorithmic Perspective on Imitation Learning16 Nov
- 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow1 Oct
- 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
Other references · 25
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Trust Region Policy Optimizationcited by 2,580
- 2017Improved Training of Wasserstein GANscited by 2,235
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
- 2017A Closer Look at Memorization in Deep Networkscited by 942
- 2016Deep Variational Information Bottleneckcited by 930
- 2015End-to-End Training of Deep Visuomotor Policiescited by 875
- 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
13 more
- 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
- 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
- 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
- 2013Black Box Variational Inferencecited by 319
- 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
- 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 204
- 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
- 2017Robust Imitation of Diverse Behaviorscited by 86
- 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
- 2017Learning From Noisy Singly-labeled Datacited by 50
- 2017Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Stepcited by 45
- 2017Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learningcited by 40
- 2017MIT Advanced Vehicle Technology Study: Large-Scale Naturalistic Driving Study of Driver Behavior and Interaction with Automationcited by 15
Built on by
Citing papers per year: 2020: 1; 2021: 2; 2022: 4; 2023: 1; 2024: 1; 2025: 2; 2026: 1.
12 papers.
- 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 51
- 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
- 2020Robust Imitation Learning from Noisy Demonstrationscited by 26
- 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 21
- 2022Disturbance-Injected Robust Imitation Learning with Task Achievementcited by 9
- 2024Consistency Matters: Defining Demonstration Data Quality Metrics in Robot Learning from Demonstrationcited by 6
- 2022Extraneousness-Aware Imitation Learningcited by 3
- 2023Imitation Learning from Purified Demonstrationscited by 3
- 2021Adaptive t-Momentum-based Optimization for Unknown Ratio of Outliers in Amateur Data in Imitation Learningcited by 2
- 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxationcited by 2
- 2025Beyond Success: Refining Elegant Robot Manipulation from Mixed-Quality Data via Just-in-Time Interventioncited by 2
- 2026VERAGMIL: Virtual Environment for Scooping Granular Foods with Imitation Learning Modelscited by 0
- 2026VERAGMIL: Virtual Environment for Scooping Granular Foods with Imitation Learning Models18 Aug 2026
- 2025Beyond Success: Refining Elegant Robot Manipulation from Mixed-Quality Data via Just-in-Time Intervention27 Nov 2025
- 2025Dexterous Manipulation through Imitation Learning: A Survey4 Apr 2025
- 2024Consistency Matters: Defining Demonstration Data Quality Metrics in Robot Learning from Demonstration18 Dec 2024
- 2023Imitation Learning from Purified Demonstrations11 Oct 2023
- 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation19 Oct 2022
- 2022Extraneousness-Aware Imitation Learning4 Oct 2022
- 2022Disturbance-Injected Robust Imitation Learning with Task Achievement9 May 2022
- 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordance5 Apr 2022
- 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality27 Oct 2021
- 2021Adaptive t-Momentum-based Optimization for Unknown Ratio of Outliers in Amateur Data in Imitation Learning2 Aug 2021
- 2020Robust Imitation Learning from Noisy Demonstrations20 Oct 2020
Similar, not cited
Close in wording, no citation either way
- 2023Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations
- 2024Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker
- 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics
- 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations
- 2023Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning
- 2025Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
- 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
- 2022Deconfounding Imitation Learning with Variational Inference