# Social-Transmotion: Promptable Human Trajectory Prediction

arXiv: https://arxiv.org/abs/2312.16168 · HTML: https://mlsanity.com/paper/2312.16168/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 55 references that are in the map.
- First posted on arXiv 26 Dec 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion](https://mlsanity.com/paper/2203.13777/) (2022)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction](https://mlsanity.com/paper/2104.00563/) (2021)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction](https://mlsanity.com/paper/1903.02793/) (2019)
- [EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning](https://mlsanity.com/paper/2303.10876/) (2023)
- [Remember Intentions: Retrospective-Memory-based Trajectory Prediction](https://mlsanity.com/paper/2203.11474/) (2022)
- [Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World](https://mlsanity.com/paper/1803.08319/) (2018)
- [Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANs](https://mlsanity.com/paper/1904.09507/) (2019)
- [CanonPose: Self-Supervised Monocular 3D Human Pose Estimation in the Wild](https://mlsanity.com/paper/2011.14679/) (2020)
- [Future Person Localization in First-Person Videos](https://mlsanity.com/paper/1711.11217/) (2017)
- [Collaborative Motion Prediction via Neural Motion Message Passing](https://mlsanity.com/paper/2003.06594/) (2020)
- [PORCA: Modeling and Planning for Autonomous Driving among Many Pedestrians](https://mlsanity.com/paper/1805.11833/) (2018)
- [Graph-based Spatial Transformer with Memory Replay for Multi-future Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2206.05712/) (2022)
- [A generic diffusion-based approach for 3D human pose prediction in the wild](https://mlsanity.com/paper/2210.05669/) (2022)
- [Are socially-aware trajectory prediction models really socially-aware?](https://mlsanity.com/paper/2108.10879/) (2021)
- [DAG-Net: Double Attentive Graph Neural Network for Trajectory Forecasting](https://mlsanity.com/paper/2005.12661/) (2020)
- [Unsupervised Sampling Promoting for Stochastic Human Trajectory Prediction](https://mlsanity.com/paper/2304.04298/) (2023)
- [Pedestrian Intention Prediction: A Multi-task Perspective](https://mlsanity.com/paper/2010.10270/) (2020)
- [Euro-PVI: Pedestrian Vehicle Interactions in Dense Urban Centers](https://mlsanity.com/paper/2106.12442/) (2021)
- [Forecasting People Trajectories and Head Poses by Jointly Reasoning on Tracklets and Vislets](https://mlsanity.com/paper/1901.02000/) (2019)

## Built on by (most cited first, 30 of 33)

- [Multi-Transmotion: Pre-trained Model for Human Motion Prediction](https://mlsanity.com/paper/2411.02673/) (2024)
- [Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2503.18589/) (2025)
- [TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sports](https://mlsanity.com/paper/2410.17785/) (2024)
- [Physical Plausibility-aware Trajectory Prediction via Locomotion Embodiment](https://mlsanity.com/paper/2503.17267/) (2025)
- [Multi-modal Knowledge Distillation-based Human Trajectory Forecasting](https://mlsanity.com/paper/2503.22201/) (2025)
- [Certified Human Trajectory Prediction](https://mlsanity.com/paper/2403.13778/) (2024)
- [Unified Human Localization and Trajectory Prediction with Monocular Vision](https://mlsanity.com/paper/2503.03535/) (2025)
- [Social-Pose: Enhancing Trajectory Prediction with Human Body Pose](https://mlsanity.com/paper/2507.22742/) (2025)
- [Waymo-3DSkelMo: A Multi-Agent 3D Skeletal Motion Dataset for Pedestrian Interaction Modeling in Autonomous Driving](https://mlsanity.com/paper/2508.09404/) (2025)
- [JointDiff: Bridging Continuous and Discrete in Multi-Agent Trajectory Generation](https://mlsanity.com/paper/2509.22522/) (2025)
- [HEADS-UP: Head-Mounted Egocentric Dataset for Trajectory Prediction in Blind Assistance Systems](https://mlsanity.com/paper/2409.20324/) (2024)
- [RealTraj: Towards Real-World Pedestrian Trajectory Forecasting](https://mlsanity.com/paper/2411.17376/) (2024)
- [OmniTraj: Pre-Training on Heterogeneous Data for Adaptive and Zero-Shot Human Trajectory Prediction](https://mlsanity.com/paper/2507.23657/) (2025)
- [EgoCogNav: Cognition-aware Human Egocentric Navigation](https://mlsanity.com/paper/2511.17581/) (2025)
- [Modeling 3D Pedestrian-Vehicle Interactions for Vehicle-Conditioned Pose Forecasting](https://mlsanity.com/paper/2602.08962/) (2026)
- [Heteroscedastic Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2605.10717/) (2026)
- [Towards Predicting Any Human Trajectory In Context](https://mlsanity.com/paper/2506.00871/) (2025)
- [HUMOF: Human Motion Forecasting in Interactive Social Scenes](https://mlsanity.com/paper/2506.03753/) (2025)
- [CrowdMAC: Masked Crowd Density Completion for Robust Crowd Density Forecasting](https://mlsanity.com/paper/2407.14725/) (2024)
- [HHI-Assist: A Dataset and Benchmark of Human-Human Interaction in Physical Assistance Scenario](https://mlsanity.com/paper/2509.10096/) (2025)
- [MMCM: Multimodality-aware Metric using Clustering-based Modes for Probabilistic Human Motion Prediction](https://mlsanity.com/paper/2511.15179/) (2025)
- [Social-Mamba: Socially-Aware Trajectory Forecasting with State-Space Models](https://mlsanity.com/paper/2605.15424/) (2026)
- [Unified Prediction and Planning via Conflict-Aware Disjoint Parameter Training](https://mlsanity.com/paper/2607.19971/) (2026)
- [Selective Social-Interaction via Individual Importance for Fast Human Trajectory Prediction](https://mlsanity.com/paper/2506.18291/) (2025)
- [PatchTraj: Unified Time-Frequency Representation Learning via Dynamic Patches for Trajectory Prediction](https://mlsanity.com/paper/2507.19119/) (2025)
- [Valeo Near-Field: a novel dataset for pedestrian intent detection](https://mlsanity.com/paper/2510.15673/) (2025)
- [Robust Human Trajectory Prediction via Self-Supervised Skeleton Representation Learning](https://mlsanity.com/paper/2602.22791/) (2026)
- [Diverse Yet Consistent: Context-Guided Diffusion with Energy-Based Joint Refinement for Multi-Agent Motion Prediction](https://mlsanity.com/paper/2605.22017/) (2026)
- [Deformable Gaussian Occupancy: Decoupling Rigid and Nonrigid Motion with Factorized Distillation](https://mlsanity.com/paper/2605.28587/) (2026)
- [Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-centric Videos](https://mlsanity.com/paper/2606.18824/) (2026)

## Abstract (arXiv)

Accurate human trajectory prediction is crucial for applications such as autonomous vehicles, robotics, and surveillance systems. Yet, existing models often fail to fully leverage the non-verbal social cues human subconsciously communicate when navigating the space. To address this, we introduce Social-Transmotion, a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior. We translate the idea of a prompt from Natural Language Processing (NLP) to the task of human trajectory prediction, where a prompt can be a sequence of x-y coordinates on the ground, bounding boxes in the image plane, or body pose keypoints in either 2D or 3D. This, in turn, augments trajectory data, leading to enhanced human trajectory prediction. Using masking technique, our model exhibits flexibility and adaptability by capturing spatiotemporal interactions between agents based on the available visual cues. We delve into the merits of using 2D versus 3D poses, and a limited set of poses. Additionally, we investigate the spatial and temporal attention map to identify which keypoints and time-steps in the sequence are vital for optimizing human trajectory prediction. Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY. The code is publicly available: https://github.com/vita-epfl/social-transmotion.