# SoMoFormer: Multi-Person Pose Forecasting with Transformers

arXiv: https://arxiv.org/abs/2208.14023 · HTML: https://mlsanity.com/paper/2208.14023/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 30 Aug 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Learning Trajectory Dependencies for Human Motion Prediction](https://mlsanity.com/paper/1908.05436/) (2019)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB](https://mlsanity.com/paper/1712.03453/) (2017)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [History Repeats Itself: Human Motion Prediction via Motion Attention](https://mlsanity.com/paper/2007.11755/) (2020)
- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [Action-Agnostic Human Pose Forecasting](https://mlsanity.com/paper/1810.09676/) (2018)
- [End-to-end Contextual Perception and Prediction with Interaction Transformer](https://mlsanity.com/paper/2008.05927/) (2020)
- [TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild](https://mlsanity.com/paper/2104.04029/) (2021)
- [Socially and Contextually Aware Human Motion and Pose Forecasting](https://mlsanity.com/paper/2007.06843/) (2020)
- [Pose Transformers (POTR): Human Motion Prediction with Non-Autoregressive Transformers](https://mlsanity.com/paper/2109.07531/) (2021)
- [Multi-level Motion Attention for Human Motion Prediction](https://mlsanity.com/paper/2106.09300/) (2021)
- [Simple Baseline for Single Human Motion Forecasting](https://mlsanity.com/paper/2110.07495/) (2021)

## Built on by (most cited first, 23 of 23)

- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment](https://mlsanity.com/paper/2403.18811/) (2024)
- [Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning](https://mlsanity.com/paper/2404.05218/) (2024)
- [A Survey on Human Interaction Motion Generation](https://mlsanity.com/paper/2503.12763/) (2025)
- [Robots That Can See: Leveraging Human Pose for Trajectory Prediction](https://mlsanity.com/paper/2309.17209/) (2023)
- [Joint-Relation Transformer for Multi-Person Motion Prediction](https://mlsanity.com/paper/2308.04808/) (2023)
- [InteRACT: Transformer Models for Human Intent Prediction Conditioned on Robot Actions](https://mlsanity.com/paper/2311.12943/) (2023)
- [Social Motion Prediction with Cognitive Hierarchies](https://mlsanity.com/paper/2311.04726/) (2023)
- [Robot Interaction Behavior Generation based on Social Motion Forecasting for Human-Robot Interaction](https://mlsanity.com/paper/2402.04768/) (2024)
- [Staged Contact-Aware Global Human Motion Forecasting](https://mlsanity.com/paper/2309.08947/) (2023)
- [ManiCast: Collaborative Manipulation with Cost-Aware Human Forecasting](https://mlsanity.com/paper/2310.13258/) (2023)
- [Efficient Multi-Person Motion Prediction by Lightweight Spatial and Temporal Interactions](https://mlsanity.com/paper/2507.09446/) (2025)
- [The MI-Motion Dataset and Benchmark for 3D Multi-Person Motion Prediction](https://mlsanity.com/paper/2306.13566/) (2023)
- [Massively Multi-Person 3D Human Motion Forecasting with Scene Context](https://mlsanity.com/paper/2409.12189/) (2024)
- [HUMOF: Human Motion Forecasting in Interactive Social Scenes](https://mlsanity.com/paper/2506.03753/) (2025)
- [PGformer: Proxy-Bridged Game Transformer for Multi-Person Highly Interactive Extreme Motion Prediction](https://mlsanity.com/paper/2306.03374/) (2023)
- [Personalized Pose Forecasting](https://mlsanity.com/paper/2312.03528/) (2023)
- [Social-MAE: Social Masked Autoencoder for Multi-person Motion Representation Learning](https://mlsanity.com/paper/2404.05578/) (2024)
- [Relation Learning and Aggregate-attention for Multi-person Motion Prediction](https://mlsanity.com/paper/2411.03729/) (2024)
- [UnityGraph: Unified Learning of Spatio-temporal features for Multi-person Motion Prediction](https://mlsanity.com/paper/2411.04151/) (2024)
- [Scriboora: Rethinking Human Pose Forecasting](https://mlsanity.com/paper/2511.15565/) (2025)
- [Modeling Fly Social Behavior with Agent-Centric Pose Forecasting](https://mlsanity.com/paper/2607.19548/) (2026)
- [Multi-Person Human Motion Forecasting in Complex Scenes](https://mlsanity.com/paper/2608.27039/) (2026)

## Abstract (arXiv)

Human pose forecasting is a challenging problem involving complex human body motion and posture dynamics. In cases that there are multiple people in the environment, one's motion may also be influenced by the motion and dynamic movements of others. Although there are several previous works targeting the problem of multi-person dynamic pose forecasting, they often model the entire pose sequence as time series (ignoring the underlying relationship between joints) or only output the future pose sequence of one person at a time. In this paper, we present a new method, called Social Motion Transformer (SoMoFormer), for multi-person 3D pose forecasting. Our transformer architecture uniquely models human motion input as a joint sequence rather than a time sequence, allowing us to perform attention over joints while predicting an entire future motion sequence for each joint in parallel. We show that with this problem reformulation, SoMoFormer naturally extends to multi-person scenes by using the joints of all people in a scene as input queries. Using learned embeddings to denote the type of joint, person identity, and global position, our model learns the relationships between joints and between people, attending more strongly to joints from the same or nearby people. SoMoFormer outperforms state-of-the-art methods for long-term motion prediction on the SoMoF benchmark as well as the CMU-Mocap and MuPoTS-3D datasets. Code will be made available after publication.