# Imitation Learning with Precisely Labeled Human Demonstrations

arXiv: https://arxiv.org/abs/2504.13803 · HTML: https://mlsanity.com/paper/2504.13803/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 18 Apr 2025.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 23 of 23)

- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Real-World Robot Learning with Masked Visual Pre-training](https://mlsanity.com/paper/2210.03109/) (2022)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [Learning to Act without Actions](https://mlsanity.com/paper/2312.10812/) (2023)
- [Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos](https://mlsanity.com/paper/2103.16817/) (2021)
- [Visual Imitation Made Easy](https://mlsanity.com/paper/2008.04899/) (2020)
- [Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations](https://mlsanity.com/paper/1912.04344/) (2019)
- [On Bringing Robots Home](https://mlsanity.com/paper/2311.16098/) (2023)
- [Learning by Watching: Physical Imitation of Manipulation Skills from Human Videos](https://mlsanity.com/paper/2101.07241/) (2021)
- [Phantom: Training Robots Without Robots Using Only Human Videos](https://mlsanity.com/paper/2503.00779/) (2025)
- [Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning](https://mlsanity.com/paper/2501.06994/) (2025)
- [Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans](https://mlsanity.com/paper/2312.00775/) (2023)
- [Vision-based Manipulation from Single Human Video with Open-World Object Graphs](https://mlsanity.com/paper/2405.20321/) (2024)
- [R+X: Retrieval and Execution from Everyday Human Videos](https://mlsanity.com/paper/2407.12957/) (2024)
- [SPOT: SE(3) Pose Trajectory Diffusion for Object-Centric Manipulation](https://mlsanity.com/paper/2411.00965/) (2024)
- [DITTO: Demonstration Imitation by Trajectory Transformation](https://mlsanity.com/paper/2403.15203/) (2024)

## Abstract (arXiv)

Within the imitation learning paradigm, training generalist robots requires large-scale datasets obtainable only through diverse curation. Due to the relative ease to collect, human demonstrations constitute a valuable addition when incorporated appropriately. However, existing methods utilizing human demonstrations face challenges in inferring precise actions, ameliorating embodiment gaps, and fusing with frontier generalist robot training pipelines. In this work, building on prior studies that demonstrate the viability of using hand-held grippers for efficient data collection, we leverage the user's control over the gripper's appearance--specifically by assigning it a unique, easily segmentable color--to enable simple and reliable application of the RANSAC and ICP registration method for precise end-effector pose estimation. We show in simulation that precisely labeled human demonstrations on their own allow policies to reach on average 88.1% of the performance of using robot demonstrations, and boost policy performance when combined with robot demonstrations, despite the inherent embodiment gap.