# Robot Interaction Behavior Generation based on Social Motion Forecasting for Human-Robot Interaction

arXiv: https://arxiv.org/abs/2402.04768 · HTML: https://mlsanity.com/paper/2402.04768/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [On human motion prediction using recurrent neural networks](https://mlsanity.com/paper/1705.02445/) (2017)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023)
- [TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis](https://mlsanity.com/paper/2305.00976/) (2023)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents](https://mlsanity.com/paper/2303.14613/) (2023)
- [Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2303.09119/) (2023)
- [Skeleton-Aware Networks for Deep Motion Retargeting](https://mlsanity.com/paper/2005.05732/) (2020)
- [MSR-GCN: Multi-Scale Residual Graph Convolution Networks for Human Motion Prediction](https://mlsanity.com/paper/2108.07152/) (2021)
- [Back to MLP: A Simple Baseline for Human Motion Prediction](https://mlsanity.com/paper/2207.01567/) (2022)
- [Space-Time-Separable Graph Convolutional Network for Pose Forecasting](https://mlsanity.com/paper/2110.04573/) (2021)
- [Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs](https://mlsanity.com/paper/1904.09507/) (2019)
- [Multi-Person Extreme Motion Prediction](https://mlsanity.com/paper/2105.08825/) (2021)
- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [Socially and Contextually Aware Human Motion and Pose Forecasting](https://mlsanity.com/paper/2007.06843/) (2020)
- [Skinned Motion Retargeting with Residual Perception of Motion Semantics & Geometry](https://mlsanity.com/paper/2303.08658/) (2023)
- [Trust-Aware Decision Making for Human-Robot Collaboration: Model Learning and Planning](https://mlsanity.com/paper/1801.04099/) (2018)
- [Contact-Aware Retargeting of Skinned Motion](https://mlsanity.com/paper/2109.07431/) (2021)
- [Trajectory-Aware Body Interaction Transformer for Multi-Person Pose Forecasting](https://mlsanity.com/paper/2303.05095/) (2023)
- [Pose Forecasting in Industrial Human-Robot Collaboration](https://mlsanity.com/paper/2208.07308/) (2022)
- [SoMoFormer: Multi-Person Pose Forecasting with Transformers](https://mlsanity.com/paper/2208.14023/) (2022)
- [Best Practices for 2-Body Pose Forecasting](https://mlsanity.com/paper/2304.05758/) (2023)
- [Self-Supervised Motion Retargeting with Safety Guarantee](https://mlsanity.com/paper/2103.06447/) (2021)
- [Robust Human Motion Forecasting using Transformer-based Model](https://mlsanity.com/paper/2302.08274/) (2023)
- [A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis](https://mlsanity.com/paper/2308.07301/) (2023)

## Built on by (most cited first, 7 of 7)

- [Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformers](https://mlsanity.com/paper/2503.13916/) (2025)
- [Learning Whole-Body Human-Humanoid Interaction from Human-Human Demonstrations](https://mlsanity.com/paper/2601.09518/) (2026)
- [RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrations](https://mlsanity.com/paper/2502.13134/) (2025)
- [Autonomous Human-Robot Interaction via Operator Imitation](https://mlsanity.com/paper/2504.02724/) (2025)
- [RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction](https://mlsanity.com/paper/2608.28693/) (2026)
- [MINT-RVAE: Multi-Cues Intention Prediction of Human-Robot Interaction using Human Pose and Emotion Information from RGB-only Camera Data](https://mlsanity.com/paper/2509.22573/) (2025)
- [Real-Time Human-Robot Interaction Intent Detection Using RGB-based Pose and Emotion Cues with Cross-Camera Model Generalization](https://mlsanity.com/paper/2512.17958/) (2025)

## Abstract (arXiv)

Integrating robots into populated environments is a complex challenge that requires an understanding of human social dynamics. In this work, we propose to model social motion forecasting in a shared human-robot representation space, which facilitates us to synthesize robot motions that interact with humans in social scenarios despite not observing any robot in the motion training. We develop a transformer-based architecture called ECHO, which operates in the aforementioned shared space to predict the future motions of the agents encountered in social scenarios. Contrary to prior works, we reformulate the social motion problem as the refinement of the predicted individual motions based on the surrounding agents, which facilitates the training while allowing for single-motion forecasting when only one human is in the scene. We evaluate our model in multi-person and human-robot motion forecasting tasks and obtain state-of-the-art performance by a large margin while being efficient and performing in real-time. Additionally, our qualitative results showcase the effectiveness of our approach in generating human-robot interaction behaviors that can be controlled via text commands. Webpage: https://evm7.github.io/ECHO/