mlsanityData to 7 Oct 2026

Imitation Learning by Estimating Expertise of Demonstrators

arXiv 2202.01288 ↗First posted 2 Feb 2022cs.LGcited by 50

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Many existing imitation learning datasets are collected from multiple demonstrators, each with different expertise at different parts of the environment. Yet, standard imitation learning algorithms typically treat all demonstrators as homogeneous, regardless of their expertise, absorbing the weaknesses of any suboptimal demonstrators. In this work, we show that unsupervised learning over demonstrator expertise can lead to a consistent boost in the performance of imitation learning algorithms. We develop and optimize a joint model over a learned policy and expertise levels of the demonstrators. This enables our model to learn from the optimal behavior and filter out the suboptimal behavior of each demonstrator. Our model learns a single policy that can outperform even the best demonstrator, and can be used to estimate the expertise of any demonstrator at any state. We illustrate our findings on real-robotic continuous control tasks from Robomimic and discrete environments such as MiniGrid and chess, out-performing competing methods in 21 out of 23 settings, with an average of 7% and up to 60% improvement in terms of the final reward.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 52 references that are in the map. First posted on arXiv 2 Feb 2022.

Builds on

29 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Data27 Oct
  2. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality27 Oct
  3. 2021Implicit Behavioral Cloning1 Sep
  4. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics10 Mar
  5. 2019Goal-conditioned Imitation Learning13 Jun

Other references · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  8. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  9. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  10. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  11. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  12. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
12 more
  1. 2017Inverse Reward Designcited by 246
  2. 2017One-Shot Imitation Learningcited by 232
  3. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  4. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  5. 2013Risk-sensitive Reinforcement Learningcited by 129
  6. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  7. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  8. 2020Aligning Superhuman AI with Human Behavior: Chess as a Model Systemcited by 78
  9. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  10. 2018Task-Embedded Control Networks for Few-Shot Imitation Learningcited by 65
  11. 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
  12. 2020Scalable Multi-Task Imitation Learning with Autonomous Improvementcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 18; 2024: 16; 2025: 7; 2026: 6.

50 papers.

Sort:

By area:

  1. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  2. 2023Data Quality in Imitation Learningcited by 72
  3. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 56
  4. 2022You Only Live Once: Single-Life Reinforcement Learningcited by 23
  5. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learningcited by 22
  6. 2022The Expertise Problem: Learning from Specialized Feedbackcited by 20
  7. 2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
  8. 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 16
  9. 2025Curating Demonstrations using Online Experiencecited by 16
  10. 2023Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learningcited by 14
  11. 2023Leveraging Demonstrations to Improve Online Learning: Quality Matterscited by 13
  12. 2023A Survey of Demonstration Learningcited by 12
38 more
  1. 2024Offline Imitation Learning Through Graph Search and Retrievalcited by 12
  2. 2023Active Policy Improvement from Multiple Black-box Oraclescited by 11
  3. 2023Scaling Laws for Imitation Learning in Single-Agent Gamescited by 11
  4. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
  5. 2023Cross-Episodic Curriculum for Transformer Agentscited by 10
  6. 2025Data Retrieval with Importance Weights for Few-Shot Imitation Learningcited by 10
  7. 2024Crowd-PrefRL: Preference-Based Reward Learning from Crowdscited by 7
  8. 2024Mamba as a motion encoder for robotic imitation learningcited by 7
  9. 2024Online Bandit Learning with Offline Preference Data for Improved RLHFcited by 6
  10. 2024Consistency Matters: Defining Demonstration Data Quality Metrics in Robot Learning from Demonstrationcited by 6
  11. 2023Bridging Imitation and Online Reinforcement Learning: An Optimistic Talecited by 5
  12. 2023Skill Disentanglement for Imitation Learning from Suboptimal Demonstrationscited by 5
  13. 2023Good Better Best: Self-Motivated Imitation Learning for noisy Demonstrationscited by 4
  14. 2024Select to Perfect: Imitating desired behavior from large multi-agent datacited by 4
  15. 2023Imitation Learning from Purified Demonstrationscited by 3
  16. 2023Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approachcited by 3
  17. 2023Modeling Boundedly Rational Agents with Latent Inference Budgetscited by 3
  18. 2024Accurate and Data-Efficient Toxicity Prediction when Annotators Disagreecited by 3
  19. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 3
  20. 2024Scalable Interactive Machine Learning for Future Command and Controlcited by 2
  21. 2025Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrationscited by 2
  22. 2025Demonstration Sidetracks: Categorizing Systematic Non-Optimality in Human Demonstrationscited by 2
  23. 2026An Efficient Metric for Data Quality Measurement in Imitation Learningcited by 2
  24. 2023I-CEE: Tailoring Explanations of Image Classification Models to User Expertisecited by 1
  25. 2024Dataset Clustering for Improved Offline Policy Learningcited by 1
  26. 2024ELA: Exploited Level Augmentation for Offline Learning in Zero-Sum Gamescited by 1
  27. 2024Learning Strategy Representation for Imitation Learning in Multi-Agent Gamescited by 1
  28. 2024Inferring Preferences from Demonstrations in Multi-objective Reinforcement Learningcited by 1
  29. 2024Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Rankercited by 1
  30. 2025Distributionally Robust Imitation Learning: Layered Control Architecture for Certifiable Autonomycited by 1
  31. 2026REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotationscited by 1
  32. 2024Boltzmann State-Dependent Rationalitycited by 0
  33. 2024Interpretable Robotic Manipulation from Languagecited by 0
  34. 2025Reinforcement Learning from Multi-level and Episodic Human Feedbackcited by 0
  35. 2026Oracle-Guided Soft Shielding for Safe Move Prediction in Chesscited by 0
  36. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approachcited by 0
  37. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0
  38. 2026Learning from Mixed-Quality Deployment Experience for Robot Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2018An Algorithmic Perspective on Imitation Learning
  2. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances
  3. 2021Imitation Learning: Progress, Taxonomies and Challenges
  4. 2021Imitation Learning by Reinforcement Learning
  5. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning
  6. 2022Robust Learning from Observation with Model Misspecification
  7. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
  8. 2026Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret