mlsanityData to 14 Oct 2026

Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality

arXiv 2110.14754 ↗First posted 27 Oct 2021cs.LG cs.AI cs.ROcited by 44

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Most existing imitation learning approaches assume the demonstrations are drawn from experts who are optimal, but relaxing this assumption enables us to use a wider range of data. Standard imitation learning may learn a suboptimal policy from demonstrations with varying optimality. Prior works use confidence scores or rankings to capture beneficial information from demonstrations with varying optimality, but they suffer from many limitations, e.g., manually annotated confidence scores or high average optimality of demonstrations. In this paper, we propose a general framework to learn from demonstrations with varying optimality that jointly learns the confidence score and a well-performing policy. Our approach, Confidence-Aware Imitation Learning (CAIL) learns a well-performing policy from confidence-reweighted demonstrations, while using an outer loss to track the performance of our model and to learn the confidence. We provide theoretical guarantees on the convergence of CAIL and evaluate its performance in both simulated and real robot experiments. Our results show that CAIL significantly outperforms other imitation learning methods from demonstrations with varying optimality. We further show that even without access to any optimal demonstrations, CAIL can still learn a successful policy, and outperforms prior work.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 16 of its 33 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

16 of 33 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics10 Mar
  2. 2020Robust Imitation Learning from Noisy Demonstrations20 Oct
  3. 2019VILD: Variational Imitation Learning with Diverse-quality Demonstrations15 Sep
  4. 2019Imitation Learning from Imperfect Demonstration27 Jan

Other references · 12

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2016OpenAI Gymcited by 1,736
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  7. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  8. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
  9. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  10. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  11. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  12. 2017OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learningcited by 40

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 11; 2024: 8; 2025: 10; 2026: 5.

44 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  3. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 58
  4. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  5. 2025Latent Diffusion Planning for Imitation Learningcited by 26
  6. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learningcited by 23
  7. 2022The Expertise Problem: Learning from Specialized Feedbackcited by 20
  8. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 19
  9. 2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
  10. 2023Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learningcited by 15
  11. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
  12. 2022Don't Copy the Teacher: Data and Model Challenges in Embodied Dialoguecited by 10
32 more
  1. 2023Guarded Policy Optimization with Imperfect Online Demonstrationscited by 10
  2. 2023Cross-Episodic Curriculum for Transformer Agentscited by 10
  3. 2024Consistency Matters: Defining Demonstration Data Quality Metrics in Robot Learning from Demonstrationcited by 6
  4. 2022Learning from Imperfect Demonstrations via Adversarial Confidence Transfercited by 5
  5. 2023Skill Disentanglement for Imitation Learning from Suboptimal Demonstrationscited by 5
  6. 2024Select to Perfect: Imitating desired behavior from large multi-agent datacited by 4
  7. 2024CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classificationcited by 4
  8. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  9. 2023Unlabeled Imperfect Demonstrations in Adversarial Imitation Learningcited by 3
  10. 2023Imitation Learning from Purified Demonstrationscited by 3
  11. 2025Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrationscited by 3
  12. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 3
  13. 2022Semi-Supervised Imitation Learning of Team Policies from Suboptimal Demonstrationscited by 2
  14. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxationcited by 2
  15. 2024DIDA: Denoised Imitation Learning based on Domain Adaptationcited by 2
  16. 2025Demonstration Sidetracks: Categorizing Systematic Non-Optimality in Human Demonstrationscited by 2
  17. 2022A Survey on Deep Reinforcement Learning-based Approaches for Adaptation and Generalizationcited by 1
  18. 2024Learning Strategy Representation for Imitation Learning in Multi-Agent Gamescited by 1
  19. 2024Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Rankercited by 1
  20. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Explorationcited by 1
  21. 2025When a Robot is More Capable than a Human: Learning from Constrained Demonstratorscited by 1
  22. 2025Restoring Noisy Demonstration for Imitation Learning With Diffusion Modelscited by 1
  23. 2026REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotationscited by 1
  24. 2023Sample-efficient Adversarial Imitation Learningcited by 0
  25. 2023Specification-Guided Data Aggregation for Semantically Aware Imitation Learningcited by 0
  26. 2024Online Policy Learning from Offline Preferencescited by 0
  27. 2025Imitation Learning via Focused Satisficingcited by 0
  28. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrationscited by 0
  29. 2025Going Beyond Expert Performance via Deep Implicit Imitation Reinforcement Learningcited by 0
  30. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approachcited by 0
  31. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0
  32. 2026RE-0: Verified Recursive Improvement of Embodied Code-as-Policy Agents through Local On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Aligning Human Intent from Imperfect Demonstrations with Confidence-based Inverse soft-Q Learning
  2. 2023Good Better Best: Self-Motivated Imitation Learning for noisy Demonstrations
  3. 2019A Divergence Minimization Perspective on Imitation Learning Methods
  4. 2025Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
  5. 2022Robust Learning from Observation with Model Misspecification
  6. 2022Robust Imitation of a Few Demonstrations with a Backwards Model
  7. 2024VITAL: Interactive Few-Shot Imitation Learning via Visual Human-in-the-Loop Corrections
  8. 2021Learning Online from Corrective Feedback: A Meta-Algorithm for Robotics