# Towards Sample-efficient Apprenticeship Learning from Suboptimal Demonstration

arXiv: https://arxiv.org/abs/2110.04347 · HTML: https://mlsanity.com/paper/2110.04347/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Placed in

- Offline RL and imitation learning from data / [Robot learning from demonstration](https://mlsanity.com/offlinerl/robot-learning-from-demonstration/)

## Builds on (most cited first, 9 of 9)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Learning from Suboptimal Demonstration via Self-Supervised Reward Regression](https://mlsanity.com/paper/2010.11723/) (2020)
- [Game-Theoretic Modeling of Human Adaptation in Human-Robot Collaboration](https://mlsanity.com/paper/1701.07790/) (2017)
- [Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillation](https://mlsanity.com/paper/2001.00503/) (2020)
- [Towards Robotic Feeding: Role of Haptics in Fork-based Food Manipulation](https://mlsanity.com/paper/1804.08768/) (2018)
- [Interpretable and Personalized Apprenticeship Scheduling: Learning Interpretable Scheduling Policies from Heterogeneous User Demonstrations](https://mlsanity.com/paper/1906.06397/) (2019)
- [Coordinated Control of UAVs for Human-Centered Active Sensing of Wildfires](https://mlsanity.com/paper/2006.07969/) (2020)
- [Measuring the Data Efficiency of Deep Learning Methods](https://mlsanity.com/paper/1907.02549/) (2019)

## Built on by (most cited first, 1 of 1)

- [Strategy Discovery and Mixture in Lifelong Learning from Heterogeneous Demonstration](https://mlsanity.com/paper/2202.07014/) (2022)

## Abstract (arXiv)

Learning from Demonstration (LfD) seeks to democratize robotics by enabling non-roboticist end-users to teach robots to perform novel tasks by providing demonstrations. However, as demonstrators are typically non-experts, modern LfD techniques are unable to produce policies much better than the suboptimal demonstration. A previously-proposed framework, SSRR, has shown success in learning from suboptimal demonstration but relies on noise-injected trajectories to infer an idealized reward function. A random approach such as noise-injection to generate trajectories has two key drawbacks: 1) Performance degradation could be random depending on whether the noise is applied to vital states and 2) Noise-injection generated trajectories may have limited suboptimality and therefore will not accurately represent the whole scope of suboptimality. We present Systematic Self-Supervised Reward Regression, S3RR, to investigate systematic alternatives for trajectory degradation. We carry out empirical evaluations and find S3RR can learn comparable or better reward correlation with ground-truth against a state-of-the-art learning from suboptimal demonstration framework.