# Learning from Imperfect Demonstrations via Adversarial Confidence Transfer

arXiv: https://arxiv.org/abs/2202.02967 · HTML: https://mlsanity.com/paper/2202.02967/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 24 of 24)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning](https://mlsanity.com/paper/1703.02949/) (2017)
- [Learning from Suboptimal Demonstration via Self-Supervised Reward Regression](https://mlsanity.com/paper/2010.11723/) (2020)
- [Wasserstein Adversarial Imitation Learning](https://mlsanity.com/paper/1906.08113/) (2019)
- [Dueling Posterior Sampling for Preference-Based Reinforcement Learning](https://mlsanity.com/paper/1908.01289/) (2019)
- [Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality](https://mlsanity.com/paper/2110.14754/) (2021)
- [DeceptionNet: Network-Driven Domain Randomization](https://mlsanity.com/paper/1904.02750/) (2019)
- [Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency](https://mlsanity.com/paper/2012.09811/) (2020)
- [Learning Multimodal Rewards from Rankings](https://mlsanity.com/paper/2109.12750/) (2021)
- [When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans](https://mlsanity.com/paper/2001.04377/) (2020)
- [Controlling Assistive Robots with Learned Latent Actions](https://mlsanity.com/paper/1909.09674/) (2019)
- [Deep Bayesian Reward Learning from Preferences](https://mlsanity.com/paper/1912.04472/) (2019)
- [Learning from Imperfect Demonstrations from Agents with Varying Dynamics](https://mlsanity.com/paper/2103.05910/) (2021)
- [Adapting Deep Visuomotor Representations with Weak Pairwise Constraints](https://mlsanity.com/paper/1511.07111/) (2015)
- [A Data-Efficient Framework for Training and Sim-to-Real Transfer of Navigation Policies](https://mlsanity.com/paper/1810.04871/) (2018)
- [Learning Feasibility to Imitate Demonstrators with Different Dynamics](https://mlsanity.com/paper/2110.15142/) (2021)
- [Domain Adaptive Imitation Learning](https://mlsanity.com/paper/1910.00105/) (2019)

## Built on by (most cited first, 5 of 5)

- [Data Quality in Imitation Learning](https://mlsanity.com/paper/2306.02437/) (2023)
- [You Only Live Once: Single-Life Reinforcement Learning](https://mlsanity.com/paper/2210.08863/) (2022)
- [What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality](https://mlsanity.com/paper/2411.13609/) (2024)
- [Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review](https://mlsanity.com/paper/2404.17687/) (2024)
- [Semi-Supervised Cross-Domain Imitation Learning](https://mlsanity.com/paper/2602.10793/) (2026)

## Abstract (arXiv)

Existing learning from demonstration algorithms usually assume access to expert demonstrations. However, this assumption is limiting in many real-world applications since the collected demonstrations may be suboptimal or even consist of failure cases. We therefore study the problem of learning from imperfect demonstrations by learning a confidence predictor. Specifically, we rely on demonstrations along with their confidence values from a different correspondent environment (source environment) to learn a confidence predictor for the environment we aim to learn a policy in (target environment -- where we only have unlabeled demonstrations.) We learn a common latent space through adversarial distribution matching of multi-length partial trajectories to enable the transfer of confidence across source and target environments. The learned confidence reweights the demonstrations to enable learning more from informative demonstrations and discarding the irrelevant ones. Our experiments in three simulated environments and a real robot reaching task demonstrate that our approach learns a policy with the highest expected return.