# Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2211.15542 · HTML: https://mlsanity.com/paper/2211.15542/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 20 of its 45 references that are in the map.
- First posted on arXiv 28 Nov 2022.

## Placed in

- Offline RL and imitation learning from data / [Robot learning from demonstration](https://mlsanity.com/offlinerl/robot-learning-from-demonstration/)

## Builds on (most cited first, 20 of 20)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences](https://mlsanity.com/paper/2006.14091/) (2020)
- [The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models](https://mlsanity.com/paper/2204.10759/) (2022)
- [Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications](https://mlsanity.com/paper/1805.07687/) (2018)
- [The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types](https://mlsanity.com/paper/2208.10687/) (2022)
- [Interactive Teaching Algorithms for Inverse Reinforcement Learning](https://mlsanity.com/paper/1905.11867/) (2019)
- [Towards Resolving Unidentifiability in Inverse Reinforcement Learning](https://mlsanity.com/paper/1601.06569/) (2016)
- [Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning](https://mlsanity.com/paper/1707.00724/) (2017)
- [Inducing Structure in Reward Learning by Learning Features](https://mlsanity.com/paper/2201.07082/) (2022)
- [I Know What You Meant: Learning Human Objectives by (Under)estimating Their Choice Set](https://mlsanity.com/paper/2011.06118/) (2020)
- [Curriculum Design for Teaching via Demonstrations: Theory and Applications](https://mlsanity.com/paper/2106.04696/) (2021)
- [Hitting the Target: Stopping Active Learning at the Cost-Based Optimum](https://mlsanity.com/paper/2110.03802/) (2021)
- [Exploring Implicit Human Responses to Robot Mistakes in a Learning from Demonstration Task](https://mlsanity.com/paper/1606.02485/) (2016)
- [Robotic Self-Assessment of Competence](https://mlsanity.com/paper/2005.01546/) (2020)

## Built on by (most cited first, 1 of 1)

- [Screw Geometry Meets Bandits: Incremental Acquisition of Demonstrations to Generate Manipulation Plans](https://mlsanity.com/paper/2410.18275/) (2024)

## Abstract (arXiv)

We examine the problem of determining demonstration sufficiency: how can a robot self-assess whether it has received enough demonstrations from an expert to ensure a desired level of performance? To address this problem, we propose a novel self-assessment approach based on Bayesian inverse reinforcement learning and value-at-risk, enabling learning-from-demonstration ("LfD") robots to compute high-confidence bounds on their performance and use these bounds to determine when they have a sufficient number of demonstrations. We propose and evaluate two definitions of sufficiency: (1) normalized expected value difference, which measures regret with respect to the human's unobserved reward function, and (2) percent improvement over a baseline policy. We demonstrate how to formulate high-confidence bounds on both of these metrics. We evaluate our approach in simulation for both discrete and continuous state-space domains and illustrate the feasibility of developing a robotic system that can accurately evaluate demonstration sufficiency. We also show that the robot can utilize active learning in asking for demonstrations from specific states which results in fewer demos needed for the robot to still maintain high confidence in its policy. Finally, via a user study, we show that our approach successfully enables robots to perform at users' desired performance levels, without needing too many or perfectly optimal demonstrations.