# Learning Actions from Human Demonstration Video for Robotic Manipulation

arXiv: https://arxiv.org/abs/1909.04312 · HTML: https://mlsanity.com/paper/1909.04312/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2019.

## Built on by (most cited first, 9 of 9)

- [VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Model](https://mlsanity.com/paper/2410.08792/) (2024)
- [Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulation](https://mlsanity.com/paper/2402.07127/) (2024)
- [ShowUI-Aloha: Human-Taught GUI Agent](https://mlsanity.com/paper/2601.07181/) (2026)
- [Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations](https://mlsanity.com/paper/2307.05959/) (2023)
- [Learn by Observation: Imitation Learning for Drone Patrolling from Videos of A Human Navigator](https://mlsanity.com/paper/2008.13193/) (2020)
- [Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraints](https://mlsanity.com/paper/2409.11518/) (2024)
- [CLIPUNetr: Assisting Human-robot Interface for Uncalibrated Visual Servoing Control with CLIP-driven Referring Expression Segmentation](https://mlsanity.com/paper/2309.09183/) (2023)
- [A Variational Graph Autoencoder for Manipulation Action Recognition and Prediction](https://mlsanity.com/paper/2110.13280/) (2021)
- [A General Mobile Manipulator Automation Framework for Flexible Manufacturing in Hostile Industrial Environments](https://mlsanity.com/paper/2302.04486/) (2023)

## Abstract (arXiv)

Learning actions from human demonstration is an emerging trend for designing intelligent robotic systems, which can be referred as video to command. The performance of such approach highly relies on the quality of video captioning. However, the general video captioning methods focus more on the understanding of the full frame, lacking of consideration on the specific object of interests in robotic manipulations. We propose a novel deep model to learn actions from human demonstration video for robotic manipulation. It consists of two deep networks, grasp detection network (GNet) and video captioning network (CNet). GNet performs two functions: providing grasp solutions and extracting the local features for the object of interests in robotic manipulation. CNet outputs the captioning results by fusing the features of both full frames and local objects. Experimental results on UR5 robotic arm show that our method could produce more accurate command from video demonstration than state-of-the-art work, thereby leading to more robust grasping performance.