# MoVEInt: Mixture of Variational Experts for Learning Human-Robot Interactions from Demonstrations

arXiv: https://arxiv.org/abs/2407.07636 · HTML: https://mlsanity.com/paper/2407.07636/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 22 of 22)

- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Learning Human-to-Robot Handovers from Point Clouds](https://mlsanity.com/paper/2303.17592/) (2023)
- [Scaling Hidden Markov Language Models](https://mlsanity.com/paper/2011.04640/) (2020)
- [Object-Independent Human-to-Robot Handovers using Real Time Robotic Vision](https://mlsanity.com/paper/2006.01797/) (2020)
- [Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration](https://mlsanity.com/paper/2108.06038/) (2021)
- [Recurrent Mixture Density Network for Spatiotemporal Visual Attention](https://mlsanity.com/paper/1603.08199/) (2016)
- [Diffusion Co-Policy for Synergistic Human-Robot Collaborative Tasks](https://mlsanity.com/paper/2305.12171/) (2023)
- [Pay attention! - Robustifying a Deep Visuomotor Policy through Task-Focused Attention](https://mlsanity.com/paper/1809.10093/) (2018)
- [Imitating by generating: deep generative models for imitation of interactive tasks](https://mlsanity.com/paper/1910.06031/) (2019)
- [Maximum Causal Tsallis Entropy Imitation Learning](https://mlsanity.com/paper/1805.08336/) (2018)
- [Perceived Safety in Physical Human Robot Interaction -- A Survey](https://mlsanity.com/paper/2105.14499/) (2021)
- [Memory-based gaze prediction in deep imitation learning for robot manipulation](https://mlsanity.com/paper/2202.04877/) (2022)
- [Hierarchical Policy Blending as Inference for Reactive Robot Control](https://mlsanity.com/paper/2210.07890/) (2022)
- [MILD: Multimodal Interactive Latent Dynamics for Learning Human-Robot Interaction](https://mlsanity.com/paper/2210.12418/) (2022)
- [Learning to Share Autonomy Across Repeated Interaction](https://mlsanity.com/paper/2107.09650/) (2021)
- [Learning Human-like Hand Reaching for Human-Robot Handshaking](https://mlsanity.com/paper/2103.00616/) (2021)
- [Adversarial Mixture Density Networks: Learning to Drive Safely from Collision Data](https://mlsanity.com/paper/2107.04485/) (2021)
- [Sample-efficient Deep Reinforcement Learning with Imaginary Rollouts for Human-Robot Interaction](https://mlsanity.com/paper/1908.05546/) (2019)
- [Maximising Coefficiency of Human-Robot Handovers through Reinforcement Learning](https://mlsanity.com/paper/2306.07205/) (2023)
- [Auto-conditioned Recurrent Mixture Density Networks for Learning Generalizable Robot Skills](https://mlsanity.com/paper/1810.00146/) (2018)
- [Hidden Markov models as recurrent neural networks: an application to Alzheimer's disease](https://mlsanity.com/paper/2006.03151/) (2020)
- [Learning Multimodal Latent Dynamics for Human-Robot Interaction](https://mlsanity.com/paper/2311.16380/) (2023)

## Built on by (most cited first, 3 of 3)

- [Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions](https://mlsanity.com/paper/2508.04681/) (2025)
- [RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrations](https://mlsanity.com/paper/2502.13134/) (2025)
- [Emergent social transmission of model-based representations without inference](https://mlsanity.com/paper/2604.05777/) (2026)

## Abstract (arXiv)

Shared dynamics models are important for capturing the complexity and variability inherent in Human-Robot Interaction (HRI). Therefore, learning such shared dynamics models can enhance coordination and adaptability to enable successful reactive interactions with a human partner. In this work, we propose a novel approach for learning a shared latent space representation for HRIs from demonstrations in a Mixture of Experts fashion for reactively generating robot actions from human observations. We train a Variational Autoencoder (VAE) to learn robot motions regularized using an informative latent space prior that captures the multimodality of the human observations via a Mixture Density Network (MDN). We show how our formulation derives from a Gaussian Mixture Regression formulation that is typically used approaches for learning HRI from demonstrations such as using an HMM/GMM for learning a joint distribution over the actions of the human and the robot. We further incorporate an additional regularization to prevent "mode collapse", a common phenomenon when using latent space mixture models with VAEs. We find that our approach of using an informative MDN prior from human observations for a VAE generates more accurate robot motions compared to previous HMM-based or recurrent approaches of learning shared latent representations, which we validate on various HRI datasets involving interactions such as handshakes, fistbumps, waving, and handovers. Further experiments in a real-world human-to-robot handover scenario show the efficacy of our approach for generating successful interactions with four different human interaction partners.