# Learning Multimodal Latent Dynamics for Human-Robot Interaction

arXiv: https://arxiv.org/abs/2311.16380 · HTML: https://mlsanity.com/paper/2311.16380/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 104 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 27 of 27)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [Learning and Retrieval from Prior Data for Skill-based Imitation Learning](https://mlsanity.com/paper/2210.11435/) (2022)
- [Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces](https://mlsanity.com/paper/1905.07357/) (2019)
- [Learning Robot Skills with Temporal Variational Inference](https://mlsanity.com/paper/2006.16232/) (2020)
- [Variational Recurrent Auto-Encoders](https://mlsanity.com/paper/1412.6581/) (2014)
- [Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies](https://mlsanity.com/paper/2112.05062/) (2021)
- [Disentangled Recurrent Wasserstein Autoencoder](https://mlsanity.com/paper/2101.07496/) (2021)
- [SKID RAW: Skill Discovery from Raw Trajectories](https://mlsanity.com/paper/2103.14610/) (2021)
- [A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models](https://mlsanity.com/paper/2209.11533/) (2022)
- [Learning Social Affordance for Human-Robot Interaction](https://mlsanity.com/paper/1604.03692/) (2016)
- [Human-robot collaboration and machine learning: a systematic review of recent research](https://mlsanity.com/paper/2110.07448/) (2021)
- [Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration](https://mlsanity.com/paper/2108.06038/) (2021)
- [Diffusion Co-Policy for Synergistic Human-Robot Collaborative Tasks](https://mlsanity.com/paper/2305.12171/) (2023)
- [Imitating by generating: deep generative models for imitation of interactive tasks](https://mlsanity.com/paper/1910.06031/) (2019)
- [Powering Hidden Markov Model by Neural Network based Generative Models](https://mlsanity.com/paper/1910.05744/) (2019)
- [MILD: Multimodal Interactive Latent Dynamics for Learning Human-Robot Interaction](https://mlsanity.com/paper/2210.12418/) (2022)
- [Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions](https://mlsanity.com/paper/1703.00503/) (2017)
- [Learning Skills from Demonstrations: A Trend from Motion Primitives to Experience Abstraction](https://mlsanity.com/paper/2210.08060/) (2022)
- [Learning Human-like Hand Reaching for Human-Robot Handshaking](https://mlsanity.com/paper/2103.00616/) (2021)
- [A Variational Time Series Feature Extractor for Action Prediction](https://mlsanity.com/paper/1807.02350/) (2018)
- [Learning Interactive Behaviors for Musculoskeletal Robots Using Bayesian Interaction Primitives](https://mlsanity.com/paper/1908.05552/) (2019)
- [Adaptation and Robust Learning of Probabilistic Movement Primitives](https://mlsanity.com/paper/1808.10648/) (2018)
- [Boosting Reinforcement Learning and Planning with Demonstrations: A Survey](https://mlsanity.com/paper/2303.13489/) (2023)

## Built on by (most cited first, 1 of 1)

- [MoVEInt: Mixture of Variational Experts for Learning Human-Robot Interactions from Demonstrations](https://mlsanity.com/paper/2407.07636/) (2024)

## Abstract (arXiv)

This article presents a method for learning well-coordinated Human-Robot Interaction (HRI) from Human-Human Interactions (HHI). We devise a hybrid approach using Hidden Markov Models (HMMs) as the latent space priors for a Variational Autoencoder to model a joint distribution over the interacting agents. We leverage the interaction dynamics learned from HHI to learn HRI and incorporate the conditional generation of robot motions from human observations into the training, thereby predicting more accurate robot trajectories. The generated robot motions are further adapted with Inverse Kinematics to ensure the desired physical proximity with a human, combining the ease of joint space learning and accurate task space reachability. For contact-rich interactions, we modulate the robot's stiffness using HMM segmentation for a compliant interaction. We verify the effectiveness of our approach deployed on a Humanoid robot via a user study. Our method generalizes well to various humans despite being trained on data from just two humans. We find that users perceive our method as more human-like, timely, and accurate and rank our method with a higher degree of preference over other baselines. We additionally show the ability of our approach to generate successful interactions in a more complex scenario of Bimanual Robot-to-Human Handovers.