# Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance

arXiv: https://arxiv.org/abs/1911.02725 · HTML: https://mlsanity.com/paper/1911.02725/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 3 of its 33 references that are in the map.
- First posted on arXiv 7 Nov 2019.

## Placed in

- Offline RL and imitation learning from data / [Robot learning from demonstration](https://mlsanity.com/offlinerl/robot-learning-from-demonstration/)

## Builds on (most cited first, 3 of 3)

- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [Kernelized Movement Primitives](https://mlsanity.com/paper/1708.08638/) (2017)

## Built on by (most cited first, 7 of 7)

- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Orientation Probabilistic Movement Primitives on Riemannian Manifolds](https://mlsanity.com/paper/2110.15036/) (2021)
- [Similarity-Aware Skill Reproduction based on Multi-Representational Learning from Demonstration](https://mlsanity.com/paper/2110.14817/) (2021)
- [Imitation Learning via Simultaneous Optimization of Policies and Auxiliary Trajectories](https://mlsanity.com/paper/2105.03019/) (2021)
- [Neural Dynamic Movement Primitives -- a survey](https://mlsanity.com/paper/2208.01903/) (2022)
- [Robot Learning from Demonstration Using Elastic Maps](https://mlsanity.com/paper/2208.02207/) (2022)
- [Transformer-XL for Long Sequence Tasks in Robotic Learning from Demonstration](https://mlsanity.com/paper/2405.15562/) (2024)

## Abstract (arXiv)

In this work, we contribute a large-scale study benchmarking the performance of multiple motion-based learning from demonstration approaches. Given the number and diversity of existing methods, it is critical that comprehensive empirical studies be performed comparing the relative strengths of these learning techniques. In particular, we evaluate four different approaches based on properties an end user may desire for real-world tasks. To perform this evaluation, we collected data from nine participants, across four different manipulation tasks with varying starting conditions. The resulting demonstrations were used to train 180 task models and evaluated on 720 task reproductions on a physical robot. Our results detail how i) complexity of the task, ii) the expertise of the human demonstrator, and iii) the starting configuration of the robot affect task performance. The collected dataset of demonstrations, robot executions, and evaluations are being made publicly available. Research insights and guidelines are also provided to guide future research and deployment choices about these approaches.