mlsanityData to 14 Oct 2026

Learning from Suboptimal Demonstration via Self-Supervised Reward Regression

arXiv 2010.11723 ↗First posted 17 Oct 2020cs.RO cs.LGcited by 84

On the map: Offline RL and imitation learning from data → Robot learning from demonstration

Abstract (arXiv)

Learning from Demonstration (LfD) seeks to democratize robotics by enabling non-roboticist end-users to teach robots to perform a task by providing a human demonstration. However, modern LfD techniques, e.g. inverse reinforcement learning (IRL), assume users provide at least stochastically optimal demonstrations. This assumption fails to hold in most real-world scenarios. Recent attempts to learn from sub-optimal demonstration leverage pairwise rankings and following the Luce-Shepard rule. However, we show these approaches make incorrect assumptions and thus suffer from brittle, degraded performance. We overcome these limitations in developing a novel approach that bootstraps off suboptimal demonstrations to synthesize optimality-parameterized data to train an idealized reward function. We empirically validate we learn an idealized reward function with ~0.95 correlation with ground-truth reward versus ~0.75 for prior work. We can then train policies achieving ~200% improvement over the suboptimal demonstration and ~90% improvement over prior work. We present a physical demonstration of teaching a robot a topspin strike in table tennis that achieves 32% faster returns and 40% more topspin than user demonstration.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 17 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 18; 2023: 15; 2024: 18; 2025: 18; 2026: 6.

84 papers.

Sort:

By area:

  1. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
  2. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  3. 2023Data Quality in Imitation Learningcited by 75
  4. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  5. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  6. 2021Learning Reward Functions from Scale Feedbackcited by 24
  7. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learningcited by 23
  8. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 21
  9. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  10. 2022The Expertise Problem: Learning from Specialized Feedbackcited by 20
  11. 2025Robot Learning from a Physical World Modelcited by 20
  12. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 19
38 more
  1. 2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrationscited by 19
  2. 2022A Ranking Game for Imitation Learningcited by 18
  3. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
  4. 2023Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learningcited by 15
  5. 2022Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrationscited by 14
  6. 2024Offline Imitation Learning Through Graph Search and Retrievalcited by 14
  7. 2023Robotic Table Tennis: A Case Study into a High Speed Learning Systemcited by 13
  8. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
  9. 2022Receding Horizon Inverse Reinforcement Learningcited by 10
  10. 2023Cross-Episodic Curriculum for Transformer Agentscited by 10
  11. 2022Disturbance-Injected Robust Imitation Learning with Task Achievementcited by 9
  12. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewardscited by 9
  13. 2025Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Modelscited by 9
  14. 2022Bayesian Disturbance Injection: Robust Imitation Learning of Flexible Policies for Robot Manipulationcited by 8
  15. 2024SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learningcited by 7
  16. 2025BOSS: Benchmark for Observation Space Shift in Long-Horizon Taskcited by 7
  17. 2025A roadmap for AI in roboticscited by 7
  18. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  19. 2022Learning from Imperfect Demonstrations via Adversarial Confidence Transfercited by 5
  20. 2023Learning Models of Adversarial Agent Behavior under Partial Observabilitycited by 5
  21. 2023Learning Reward for Physical Skills using Large Language Modelcited by 5
  22. 2024Learning Diverse Robot Striking Motions with Diffusion Models and Kinematically Constrained Gradient Guidancecited by 5
  23. 2024ARCADE: Scalable Demonstration Collection and Generation via Augmented Reality for Imitation Learningcited by 5
  24. 2021Natural Language Specification of Reinforcement Learning Policies through Differentiable Decision Treescited by 4
  25. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Rewardcited by 4
  26. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  27. 2024Multi-Camera Asynchronous Ball Localization and Trajectory Prediction with Factor Graphs and Human Posescited by 4
  28. 2024Select to Perfect: Imitating desired behavior from large multi-agent datacited by 4
  29. 2021Inverse reinforcement learning for autonomous navigation via differentiable semantic mapping and planningcited by 3
  30. 2021Marginal MAP Estimation for Inverse RL under Occlusion with Observer Noisecited by 3
  31. 2021Back to Reality for Imitation Learningcited by 3
  32. 2022Safe Inverse Reinforcement Learning via Control Barrier Functioncited by 3
  33. 2023Unlabeled Imperfect Demonstrations in Adversarial Imitation Learningcited by 3
  34. 2023Imitation Learning from Purified Demonstrationscited by 3
  35. 2024CANDERE-COACH: Reinforcement Learning from Noisy Feedbackcited by 3
  36. 2025Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitationcited by 3
  37. 2025Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrationscited by 3
  38. 2025Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AIcited by 3

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning
  2. 2021Learning from Successful and Failed Demonstrations via Optimization
  3. 2023A Survey of Demonstration Learning
  4. 2018Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications
  5. 2024Reward Learning from Suboptimal Demonstrations with Applications in Surgical Electrocautery
  6. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles
  7. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferences
  8. 2018Teaching Inverse Reinforcement Learners via Features and Demonstrations