# Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality

arXiv: https://arxiv.org/abs/2110.14754 · HTML: https://mlsanity.com/paper/2110.14754/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 16 of 16)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Learning from Suboptimal Demonstration via Self-Supervised Reward Regression](https://mlsanity.com/paper/2010.11723/) (2020)
- [Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences](https://mlsanity.com/paper/2006.14091/) (2020)
- [Dueling Posterior Sampling for Preference-Based Reinforcement Learning](https://mlsanity.com/paper/1908.01289/) (2019)
- [OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1709.06683/) (2017)
- [Learning from Imperfect Demonstrations from Agents with Varying Dynamics](https://mlsanity.com/paper/2103.05910/) (2021)
- [Robust Imitation Learning from Noisy Demonstrations](https://mlsanity.com/paper/2010.10181/) (2020)
- [VILD: Variational Imitation Learning with Diverse-quality Demonstrations](https://mlsanity.com/paper/1909.06769/) (2019)

## Built on by (most cited first, 30 of 44)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2207.10050/) (2022)
- [Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets](https://mlsanity.com/paper/2304.08742/) (2023)
- [Imitation Learning by Estimating Expertise of Demonstrators](https://mlsanity.com/paper/2202.01288/) (2022)
- [Latent Diffusion Planning for Imitation Learning](https://mlsanity.com/paper/2504.16925/) (2025)
- [Eliciting Compatible Demonstrations for Multi-Human Imitation Learning](https://mlsanity.com/paper/2210.08073/) (2022)
- [The Expertise Problem: Learning from Specialized Feedback](https://mlsanity.com/paper/2211.06519/) (2022)
- [Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction](https://mlsanity.com/paper/2207.03395/) (2022)
- [Distance Weighted Supervised Learning for Offline Interaction Data](https://mlsanity.com/paper/2304.13774/) (2023)
- [Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning](https://mlsanity.com/paper/2310.14196/) (2023)
- [Inverse Reinforcement Learning by Estimating Expertise of Demonstrators](https://mlsanity.com/paper/2402.01886/) (2024)
- [Don't Copy the Teacher: Data and Model Challenges in Embodied Dialogue](https://mlsanity.com/paper/2210.04443/) (2022)
- [Guarded Policy Optimization with Imperfect Online Demonstrations](https://mlsanity.com/paper/2303.01728/) (2023)
- [Cross-Episodic Curriculum for Transformer Agents](https://mlsanity.com/paper/2310.08549/) (2023)
- [Consistency Matters: Defining Demonstration Data Quality Metrics in Robot Learning from Demonstration](https://mlsanity.com/paper/2412.14309/) (2024)
- [Learning from Imperfect Demonstrations via Adversarial Confidence Transfer](https://mlsanity.com/paper/2202.02967/) (2022)
- [Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2306.07919/) (2023)
- [Select to Perfect: Imitating desired behavior from large multi-agent data](https://mlsanity.com/paper/2405.03735/) (2024)
- [CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification](https://mlsanity.com/paper/2410.03038/) (2024)
- [Policy Learning from Large Vision-Language Model Feedback without Reward Modeling](https://mlsanity.com/paper/2507.23391/) (2025)
- [Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning](https://mlsanity.com/paper/2302.06271/) (2023)
- [Imitation Learning from Purified Demonstrations](https://mlsanity.com/paper/2310.07143/) (2023)
- [Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrations](https://mlsanity.com/paper/2505.10760/) (2025)
- [Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting](https://mlsanity.com/paper/2603.08118/) (2026)
- [Semi-Supervised Imitation Learning of Team Policies from Suboptimal Demonstrations](https://mlsanity.com/paper/2205.02959/) (2022)
- [Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation](https://mlsanity.com/paper/2210.10469/) (2022)
- [DIDA: Denoised Imitation Learning based on Domain Adaptation](https://mlsanity.com/paper/2404.03382/) (2024)
- [Demonstration Sidetracks: Categorizing Systematic Non-Optimality in Human Demonstrations](https://mlsanity.com/paper/2506.11262/) (2025)
- [A Survey on Deep Reinforcement Learning-based Approaches for Adaptation and Generalization](https://mlsanity.com/paper/2202.08444/) (2022)
- [Learning Strategy Representation for Imitation Learning in Multi-Agent Games](https://mlsanity.com/paper/2409.19363/) (2024)

## Abstract (arXiv)

Most existing imitation learning approaches assume the demonstrations are drawn from experts who are optimal, but relaxing this assumption enables us to use a wider range of data. Standard imitation learning may learn a suboptimal policy from demonstrations with varying optimality. Prior works use confidence scores or rankings to capture beneficial information from demonstrations with varying optimality, but they suffer from many limitations, e.g., manually annotated confidence scores or high average optimality of demonstrations. In this paper, we propose a general framework to learn from demonstrations with varying optimality that jointly learns the confidence score and a well-performing policy. Our approach, Confidence-Aware Imitation Learning (CAIL) learns a well-performing policy from confidence-reweighted demonstrations, while using an outer loss to track the performance of our model and to learn the confidence. We provide theoretical guarantees on the convergence of CAIL and evaluate its performance in both simulated and real robot experiments. Our results show that CAIL significantly outperforms other imitation learning methods from demonstrations with varying optimality. We further show that even without access to any optimal demonstrations, CAIL can still learn a successful policy, and outperforms prior work.