# Robotic Imitation of Human Actions

arXiv: https://arxiv.org/abs/2401.08381 · HTML: https://mlsanity.com/paper/2401.08381/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [Diffusion Action Segmentation](https://mlsanity.com/paper/2303.17959/) (2023)
- [NICOL: A Neuro-inspired Collaborative Semi-humanoid Robot that Bridges Social Interaction and Reliable Manipulation](https://mlsanity.com/paper/2305.08528/) (2023)
- [Semantic2Graph: Graph-based Multi-modal Feature Fusion for Action Segmentation in Videos](https://mlsanity.com/paper/2209.05653/) (2022)
- [Learning Reusable Manipulation Strategies](https://mlsanity.com/paper/2311.03293/) (2023)
- [Predictive Coding-based Deep Dynamic Neural Network for Visuomotor Learning](https://mlsanity.com/paper/1706.02444/) (2017)
- [Decoupling Skill Learning from Robotic Control for Generalizable Object Manipulation](https://mlsanity.com/paper/2303.04016/) (2023)

## Built on by (most cited first, 2 of 2)

- [Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models](https://mlsanity.com/paper/2408.07975/) (2024)
- [DIRIGENt: End-To-End Robotic Imitation of Human Demonstrations Based on a Diffusion Model](https://mlsanity.com/paper/2501.16800/) (2025)

## Abstract (arXiv)

Imitation can allow us to quickly gain an understanding of a new task. Through a demonstration, we can gain direct knowledge about which actions need to be performed and which goals they have. In this paper, we introduce a new approach to imitation learning that tackles the challenges of a robot imitating a human, such as the change in perspective and body schema. Our approach can use a single human demonstration to abstract information about the demonstrated task, and use that information to generalise and replicate it. We facilitate this ability by a new integration of two state-of-the-art methods: a diffusion action segmentation model to abstract temporal information from the demonstration and an open vocabulary object detector for spatial information. Furthermore, we refine the abstracted information and use symbolic reasoning to create an action plan utilising inverse kinematics, to allow the robot to imitate the demonstrated action.