# Consistency Matters: Defining Demonstration Data Quality Metrics in Robot Learning from Demonstration

arXiv: https://arxiv.org/abs/2412.14309 · HTML: https://mlsanity.com/paper/2412.14309/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 11 of its 46 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot learning from demonstration](https://mlsanity.com/offlinerl/robot-learning-from-demonstration/)

## Builds on (most cited first, 11 of 11)

- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Robust Imitation of Diverse Behaviors](https://mlsanity.com/paper/1707.02747/) (2017)
- [Imitation Learning by Estimating Expertise of Demonstrators](https://mlsanity.com/paper/2202.01288/) (2022)
- [Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality](https://mlsanity.com/paper/2110.14754/) (2021)
- [A survey of robot learning from demonstrations for Human-Robot Collaboration](https://mlsanity.com/paper/1710.08789/) (2017)
- [Eliciting Compatible Demonstrations for Multi-Human Imitation Learning](https://mlsanity.com/paper/2210.08073/) (2022)
- [VILD: Variational Imitation Learning with Diverse-quality Demonstrations](https://mlsanity.com/paper/1909.06769/) (2019)
- [Improving User Specifications for Robot Behavior through Active Preference Learning: Framework and Evaluation](https://mlsanity.com/paper/1907.10412/) (2019)
- [Quantifying Demonstration Quality for Robot Learning and Generalization](https://mlsanity.com/paper/2203.13843/) (2022)
- [A new approach to evaluating legibility: Comparing legibility frameworks using framework-independent robot motion trajectories](https://mlsanity.com/paper/2201.05765/) (2022)

## Built on by (most cited first, 6 of 6)

- [Learning from the Best: Smoothness-Driven Metrics for Data Quality in Imitation Learning](https://mlsanity.com/paper/2604.23000/) (2026)
- [An Efficient Metric for Data Quality Measurement in Imitation Learning](https://mlsanity.com/paper/2605.01544/) (2026)
- [Robots That Know What to Ask: Recovering Misaligned Rewards through Targeted Explanations](https://mlsanity.com/paper/2605.22986/) (2026)
- [Bidirectional Tutoring for Developmental Motor Learning in Robots: Co-Developed Interaction Dynamics Support Stable Learning](https://mlsanity.com/paper/2606.19728/) (2026)
- [Auditing Instruction-Trajectory Mismatches in Multimodal Robot Demonstrations](https://mlsanity.com/paper/2608.07895/) (2026)
- [BlenDAgger: Blended Shared Control for Interactive Imitation Learning](https://mlsanity.com/paper/2609.37599/) (2026)

## Abstract (arXiv)

Learning from Demonstration (LfD) empowers robots to acquire new skills through human demonstrations, making it feasible for everyday users to teach robots. However, the success of learning and generalization heavily depends on the quality of these demonstrations. Consistency is often used to indicate quality in LfD, yet the factors that define this consistency remain underexplored. In this paper, we evaluate a comprehensive set of motion data characteristics to determine which consistency measures best predict learning performance. By ensuring demonstration consistency prior to training, we enhance models' predictive accuracy and generalization to novel scenarios. We validate our approach with two user studies involving participants with diverse levels of robotics expertise. In the first study (N = 24), users taught a PR2 robot to perform a button-pressing task in a constrained environment, while in the second study (N = 30), participants trained a UR5 robot on a pick-and-place task. Results show that demonstration consistency significantly impacts success rates in both learning and generalization, with 70% and 89% of task success rates in the two studies predicted using our consistency metrics. Moreover, our metrics estimate generalized performance success rates with 76% and 91% accuracy. These findings suggest that our proposed measures provide an intuitive, practical way to assess demonstration data quality before training, without requiring expert data or algorithm-specific modifications. Our approach offers a systematic way to evaluate demonstration quality, addressing a critical gap in LfD by formalizing consistency metrics that enhance the reliability of robot learning from human demonstrations.