# Imitating by generating: deep generative models for imitation of interactive tasks

arXiv: https://arxiv.org/abs/1910.06031 · HTML: https://mlsanity.com/paper/1910.06031/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2019.

## Built on by (most cited first, 11 of 11)

- [Bayesian Meta-Learning for Few-Shot Policy Adaptation Across Robotic Platforms](https://mlsanity.com/paper/2103.03697/) (2021)
- [Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions](https://mlsanity.com/paper/2508.04681/) (2025)
- [MILD: Multimodal Interactive Latent Dynamics for Learning Human-Robot Interaction](https://mlsanity.com/paper/2210.12418/) (2022)
- [RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrations](https://mlsanity.com/paper/2502.13134/) (2025)
- [MoVEInt: Mixture of Variational Experts for Learning Human-Robot Interactions from Demonstrations](https://mlsanity.com/paper/2407.07636/) (2024)
- [Autonomous Human-Robot Interaction via Operator Imitation](https://mlsanity.com/paper/2504.02724/) (2025)
- [Improving Human Motion Prediction Through Continual Learning](https://mlsanity.com/paper/2107.00544/) (2021)
- [Synthesis and Execution of Communicative Robotic Movements with Generative Adversarial Networks](https://mlsanity.com/paper/2203.15640/) (2022)
- [Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models](https://mlsanity.com/paper/2204.08573/) (2022)
- [Learning Multimodal Latent Dynamics for Human-Robot Interaction](https://mlsanity.com/paper/2311.16380/) (2023)
- [Robotic Assistant: Completing Collaborative Tasks with Dexterous Vision-Language-Action Models](https://mlsanity.com/paper/2510.25713/) (2025)

## Abstract (arXiv)

To coordinate actions with an interaction partner requires a constant exchange of sensorimotor signals. Humans acquire these skills in infancy and early childhood mostly by imitation learning and active engagement with a skilled partner. They require the ability to predict and adapt to one's partner during an interaction. In this work we want to explore these ideas in a human-robot interaction setting in which a robot is required to learn interactive tasks from a combination of observational and kinesthetic learning. To this end, we propose a deep learning framework consisting of a number of components for (1) human and robot motion embedding, (2) motion prediction of the human partner and (3) generation of robot joint trajectories matching the human motion. To test these ideas, we collect human-human interaction data and human-robot interaction data of four interactive tasks "hand-shake", "hand-wave", "parachute fist-bump" and "rocket fist-bump". We demonstrate experimentally the importance of predictive and adaptive components as well as low-level abstractions to successfully learn to imitate human behavior in interactive social tasks.