# Joint-Relation Transformer for Multi-Person Motion Prediction

arXiv: https://arxiv.org/abs/2308.04808 · HTML: https://mlsanity.com/paper/2308.04808/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 9 Aug 2023.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [Rethinking Graph Transformers with Spectral Attention](https://mlsanity.com/paper/2106.03893/) (2021)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB](https://mlsanity.com/paper/1712.03453/) (2017)
- [Representing Long-Range Context for Graph Neural Networks with Global Attention](https://mlsanity.com/paper/2201.08821/) (2022)
- [Graph-Bert: Only Attention is Needed for Learning Graph Representations](https://mlsanity.com/paper/2001.05140/) (2020)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [GraphiT: Encoding Graph Structure in Transformers](https://mlsanity.com/paper/2106.05667/) (2021)
- [Dynamic Multiscale Graph Neural Networks for 3D Skeleton-Based Human Motion Prediction](https://mlsanity.com/paper/2003.08802/) (2020)
- [GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning](https://mlsanity.com/paper/2204.08770/) (2022)
- [Convolutional Sequence to Sequence Model for Human Dynamics](https://mlsanity.com/paper/1805.00655/) (2018)
- [QuaterNet: A Quaternion-based Recurrent Model for Human Motion](https://mlsanity.com/paper/1805.06485/) (2018)
- [Deep representation learning for human motion prediction and classification](https://mlsanity.com/paper/1702.07486/) (2017)
- [EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning](https://mlsanity.com/paper/2303.10876/) (2023)
- [A Neural Temporal Model for Human Motion Prediction](https://mlsanity.com/paper/1809.03036/) (2018)
- [Remember Intentions: Retrospective-Memory-based Trajectory Prediction](https://mlsanity.com/paper/2203.11474/) (2022)
- [Multi-Person Extreme Motion Prediction](https://mlsanity.com/paper/2105.08825/) (2021)
- [Learning Human Motion Models for Long-term Predictions](https://mlsanity.com/paper/1704.02827/) (2017)
- [Gophormer: Ego-Graph Transformer for Node Classification](https://mlsanity.com/paper/2110.13094/) (2021)
- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [Action-Agnostic Human Pose Forecasting](https://mlsanity.com/paper/1810.09676/) (2018)
- [TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild](https://mlsanity.com/paper/2104.04029/) (2021)
- [Socially and Contextually Aware Human Motion and Pose Forecasting](https://mlsanity.com/paper/2007.06843/) (2020)

## Built on by (most cited first, 16 of 16)

- [Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning](https://mlsanity.com/paper/2404.05218/) (2024)
- [Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption](https://mlsanity.com/paper/2404.11291/) (2024)
- [Language-Driven Interactive Traffic Trajectory Generation](https://mlsanity.com/paper/2405.15388/) (2024)
- [Ponimator: Unfolding Interactive Pose for Versatile Human-human Interaction Animation](https://mlsanity.com/paper/2510.14976/) (2025)
- [Human Motion Prediction under Unexpected Perturbation](https://mlsanity.com/paper/2403.15891/) (2024)
- [Orientation-Aware Leg Movement Learning for Action-Driven Human Motion Prediction](https://mlsanity.com/paper/2310.14907/) (2023)
- [Large-Scale Multi-Character Interaction Synthesis](https://mlsanity.com/paper/2505.14087/) (2025)
- [Efficient Multi-Person Motion Prediction by Lightweight Spatial and Temporal Interactions](https://mlsanity.com/paper/2507.09446/) (2025)
- [HUMOF: Human Motion Forecasting in Interactive Social Scenes](https://mlsanity.com/paper/2506.03753/) (2025)
- [Ego-Human Motion Prediction with 3D-Aware LLM](https://mlsanity.com/paper/2607.07001/) (2026)
- [Relation Learning and Aggregate-attention for Multi-person Motion Prediction](https://mlsanity.com/paper/2411.03729/) (2024)
- [UnityGraph: Unified Learning of Spatio-temporal features for Multi-person Motion Prediction](https://mlsanity.com/paper/2411.04151/) (2024)
- [Scriboora: Rethinking Human Pose Forecasting](https://mlsanity.com/paper/2511.15565/) (2025)
- [Spatiotemporal-Untrammelled Mixture of Experts for Multi-Person Motion Prediction](https://mlsanity.com/paper/2512.21707/) (2025)
- [Residual Flow Matching with Dynamic Cross-Interaction for 3D Multi-Person Motion Prediction](https://mlsanity.com/paper/2608.03379/) (2026)
- [Multi-Person Human Motion Forecasting in Complex Scenes](https://mlsanity.com/paper/2608.27039/) (2026)

## Abstract (arXiv)

Multi-person motion prediction is a challenging problem due to the dependency of motion on both individual past movements and interactions with other people. Transformer-based methods have shown promising results on this task, but they miss the explicit relation representation between joints, such as skeleton structure and pairwise distance, which is crucial for accurate interaction modeling. In this paper, we propose the Joint-Relation Transformer, which utilizes relation information to enhance interaction modeling and improve future motion prediction. Our relation information contains the relative distance and the intra-/inter-person physical constraints. To fuse relation and joint information, we design a novel joint-relation fusion layer with relation-aware attention to update both features. Additionally, we supervise the relation information by forecasting future distance. Experiments show that our method achieves a 13.4% improvement of 900ms VIM on 3DPW-SoMoF/RC and 17.8%/12.0% improvement of 3s MPJPE on CMU-Mpcap/MuPoTS-3D dataset.