# Learning from Imperfect Demonstrations from Agents with Varying Dynamics

arXiv: https://arxiv.org/abs/2103.05910 · HTML: https://mlsanity.com/paper/2103.05910/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 10 Mar 2021.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Importance Weighted Adversarial Nets for Partial Domain Adaptation](https://mlsanity.com/paper/1803.09210/) (2018)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [State Alignment-based Imitation Learning](https://mlsanity.com/paper/1911.10947/) (2019)
- [Wasserstein Adversarial Imitation Learning](https://mlsanity.com/paper/1906.08113/) (2019)
- [Adversarial Imitation via Variational Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.06404/) (2018)
- [When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans](https://mlsanity.com/paper/2001.04377/) (2020)
- [Controlling Assistive Robots with Learned Latent Actions](https://mlsanity.com/paper/1909.09674/) (2019)
- [Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving](https://mlsanity.com/paper/2007.00178/) (2020)
- [A First Look at Ad-block Detection: A New Arms Race on the Web](https://mlsanity.com/paper/1605.05841/) (2016)
- [Learning Sparse Rewarded Tasks from Sub-Optimal Demonstrations](https://mlsanity.com/paper/2004.00530/) (2020)
- [Continuous Online Learning and New Insights to Online Imitation Learning](https://mlsanity.com/paper/1912.01261/) (2019)

## Built on by (most cited first, 27 of 27)

- [Learning Generalizable Dexterous Manipulation from Human Grasp Affordance](https://mlsanity.com/paper/2204.02320/) (2022)
- [Imitation Learning by Estimating Expertise of Demonstrators](https://mlsanity.com/paper/2202.01288/) (2022)
- [Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality](https://mlsanity.com/paper/2110.14754/) (2021)
- [The Expertise Problem: Learning from Specialized Feedback](https://mlsanity.com/paper/2211.06519/) (2022)
- [MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations](https://mlsanity.com/paper/2303.17156/) (2023)
- [Distance Weighted Supervised Learning for Offline Interaction Data](https://mlsanity.com/paper/2304.13774/) (2023)
- [A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agents](https://mlsanity.com/paper/2402.04580/) (2024)
- [Inverse Reinforcement Learning by Estimating Expertise of Demonstrators](https://mlsanity.com/paper/2402.01886/) (2024)
- [Learning Feasibility to Imitate Demonstrators with Different Dynamics](https://mlsanity.com/paper/2110.15142/) (2021)
- [Cross-Episodic Curriculum for Transformer Agents](https://mlsanity.com/paper/2310.08549/) (2023)
- [Learning Preferences for Interactive Autonomy](https://mlsanity.com/paper/2210.10899/) (2022)
- [Learning from Imperfect Demonstrations via Adversarial Confidence Transfer](https://mlsanity.com/paper/2202.02967/) (2022)
- [IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic](https://mlsanity.com/paper/2502.19859/) (2025)
- [Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward](https://mlsanity.com/paper/2212.01509/) (2022)
- [Select to Perfect: Imitating desired behavior from large multi-agent data](https://mlsanity.com/paper/2405.03735/) (2024)
- [Out-of-Dynamics Imitation Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2211.06839/) (2022)
- [Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitation](https://mlsanity.com/paper/2501.01367/) (2025)
- [Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation](https://mlsanity.com/paper/2210.10469/) (2022)
- [InterPReT: Interactive Policy Restructuring and Training Enable Effective Imitation Learning from Laypersons](https://mlsanity.com/paper/2602.04213/) (2026)
- [Flip Stunts on Bicycle Robots using Iterative Motion Imitation](https://mlsanity.com/paper/2603.27944/) (2026)
- [Feasibility-aware Imitation Learning from Observations through a Hand-mounted Demonstration Interface](https://mlsanity.com/paper/2503.09018/) (2025)
- [Flexible Locomotion Learning with Diffusion Model Predictive Control](https://mlsanity.com/paper/2510.04234/) (2025)
- [Task-Agnostic Learning to Accomplish New Tasks](https://mlsanity.com/paper/2209.04100/) (2022)
- [Divide and Repair: Using Options to Improve Performance of Imitation Learning Against Adversarial Demonstrations](https://mlsanity.com/paper/2306.04581/) (2023)
- [Learning from Observation: A Survey of Recent Advances](https://mlsanity.com/paper/2509.19379/) (2025)
- [Feasibility-aware Imitation Learning from Observation with Multimodal Feedback](https://mlsanity.com/paper/2602.15351/) (2026)
- [Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2607.09866/) (2026)

## Abstract (arXiv)

Imitation learning enables robots to learn from demonstrations. Previous imitation learning algorithms usually assume access to optimal expert demonstrations. However, in many real-world applications, this assumption is limiting. Most collected demonstrations are not optimal or are produced by an agent with slightly different dynamics. We therefore address the problem of imitation learning when the demonstrations can be sub-optimal or be drawn from agents with varying dynamics. We develop a metric composed of a feasibility score and an optimality score to measure how useful a demonstration is for imitation learning. The proposed score enables learning from more informative demonstrations, and disregarding the less relevant demonstrations. Our experiments on four environments in simulation and on a real robot show improved learned policies with higher expected return.