mlsanityData to 14 Oct 2026

Social-Transmotion: Promptable Human Trajectory Prediction

arXiv 2312.16168 ↗First posted 26 Dec 2023cs.CV cs.ROcited by 33

Not on the map. Why?

Abstract (arXiv)

Accurate human trajectory prediction is crucial for applications such as autonomous vehicles, robotics, and surveillance systems. Yet, existing models often fail to fully leverage the non-verbal social cues human subconsciously communicate when navigating the space. To address this, we introduce Social-Transmotion, a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior. We translate the idea of a prompt from Natural Language Processing (NLP) to the task of human trajectory prediction, where a prompt can be a sequence of x-y coordinates on the ground, bounding boxes in the image plane, or body pose keypoints in either 2D or 3D. This, in turn, augments trajectory data, leading to enhanced human trajectory prediction. Using masking technique, our model exhibits flexibility and adaptability by capturing spatiotemporal interactions between agents based on the available visual cues. We delve into the merits of using 2D versus 3D poses, and a limited set of poses. Additionally, we investigate the spatial and temporal attention map to identify which keypoints and time-steps in the sequence are vital for optimizing human trajectory prediction. Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY. The code is publicly available: https://github.com/vita-epfl/social-transmotion.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 26 Dec 2023.

Builds on

36 of 55 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  3. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 690
  4. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 331
  5. 2018SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraintscited by 298
  6. 2020Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Predictioncited by 218
  7. 2022Stochastic Trajectory Prediction via Motion Indeterminacy Diffusioncited by 185
  8. 2020Transformer Networks for Trajectory Forecastingcited by 181
  9. 2018Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learningcited by 175
  10. 2020From Goals, Waypoints & Paths To Long Term Human Trajectory Forecastingcited by 167
  11. 2021Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Predictioncited by 131
  12. 2020Human Trajectory Forecasting in Crowds: A Deep Learning Perspectivecited by 129
24 more
  1. 2019Peeking into the Future: Predicting Future Person Activities and Locations in Videoscited by 114
  2. 2019SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Predictioncited by 111
  3. 2023EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoningcited by 97
  4. 2022Remember Intentions: Retrospective-Memory-based Trajectory Predictioncited by 86
  5. 2018Learning to Detect and Track Visible and Occluded Body Joints in a Virtual Worldcited by 82
  6. 2019Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories with GANscited by 79
  7. 2020CanonPose: Self-Supervised Monocular 3D Human Pose Estimation in the Wildcited by 65
  8. 2017Future Person Localization in First-Person Videoscited by 58
  9. 2020Collaborative Motion Prediction via Neural Motion Message Passingcited by 55
  10. 2018PORCA: Modeling and Planning for Autonomous Driving among Many Pedestrianscited by 36
  11. 2022Graph-based Spatial Transformer with Memory Replay for Multi-future Pedestrian Trajectory Predictioncited by 36
  12. 2022A generic diffusion-based approach for 3D human pose prediction in the wildcited by 33
  13. 2021Are socially-aware trajectory prediction models really socially-aware?cited by 32
  14. 2020DAG-Net: Double Attentive Graph Neural Network for Trajectory Forecastingcited by 30
  15. 2023Unsupervised Sampling Promoting for Stochastic Human Trajectory Predictioncited by 17
  16. 2020Pedestrian Intention Prediction: A Multi-task Perspectivecited by 16
  17. 2021Euro-PVI: Pedestrian Vehicle Interactions in Dense Urban Centerscited by 16
  18. 2019Forecasting People Trajectories and Head Poses by Jointly Reasoning on Tracklets and Visletscited by 13
  19. 2022BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimationcited by 13
  20. 2022Pedestrian 3D Bounding Box Predictioncited by 11
  21. 2023Toward Reliable Human Pose Forecasting with Uncertaintycited by 11
  22. 2022Learning the Pedestrian-Vehicle Interaction for Pedestrian Trajectory Predictioncited by 7
  23. 2021Pose and Semantic Map Based Probabilistic Forecast of Vulnerable Road Users' Trajectoriescited by 6
  24. 2021A Shared Representation for Photorealistic Driving Simulatorscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 16; 2026: 11.

33 papers.

Sort:

By area:

  1. 2024Multi-Transmotion: Pre-trained Model for Human Motion Predictioncited by 21
  2. 2025Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modelingcited by 15
  3. 2024TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sportscited by 11
  4. 2025Physical Plausibility-aware Trajectory Prediction via Locomotion Embodimentcited by 10
  5. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  6. 2024Certified Human Trajectory Predictioncited by 6
  7. 2025Unified Human Localization and Trajectory Prediction with Monocular Visioncited by 5
  8. 2025Social-Pose: Enhancing Trajectory Prediction with Human Body Posecited by 5
  9. 2025Waymo-3DSkelMo: A Multi-Agent 3D Skeletal Motion Dataset for Pedestrian Interaction Modeling in Autonomous Drivingcited by 5
  10. 2025JointDiff: Bridging Continuous and Discrete in Multi-Agent Trajectory Generationcited by 5
  11. 2024HEADS-UP: Head-Mounted Egocentric Dataset for Trajectory Prediction in Blind Assistance Systemscited by 3
  12. 2024RealTraj: Towards Real-World Pedestrian Trajectory Forecastingcited by 3
21 more
  1. 2025OmniTraj: Pre-Training on Heterogeneous Data for Adaptive and Zero-Shot Human Trajectory Predictioncited by 3
  2. 2025EgoCogNav: Cognition-aware Human Egocentric Navigationcited by 3
  3. 2026Modeling 3D Pedestrian-Vehicle Interactions for Vehicle-Conditioned Pose Forecastingcited by 3
  4. 2026Heteroscedastic Diffusion for Multi-Agent Trajectory Modelingcited by 3
  5. 2025Towards Predicting Any Human Trajectory In Contextcited by 2
  6. 2025HUMOF: Human Motion Forecasting in Interactive Social Scenescited by 2
  7. 2024CrowdMAC: Masked Crowd Density Completion for Robust Crowd Density Forecastingcited by 1
  8. 2025HHI-Assist: A Dataset and Benchmark of Human-Human Interaction in Physical Assistance Scenariocited by 1
  9. 2025MMCM: Multimodality-aware Metric using Clustering-based Modes for Probabilistic Human Motion Predictioncited by 1
  10. 2026Social-Mamba: Socially-Aware Trajectory Forecasting with State-Space Modelscited by 1
  11. 2026Unified Prediction and Planning via Conflict-Aware Disjoint Parameter Trainingcited by 1
  12. 2025Selective Social-Interaction via Individual Importance for Fast Human Trajectory Predictioncited by 0
  13. 2025PatchTraj: Unified Time-Frequency Representation Learning via Dynamic Patches for Trajectory Predictioncited by 0
  14. 2025Valeo Near-Field: a novel dataset for pedestrian intent detectioncited by 0
  15. 2026Robust Human Trajectory Prediction via Self-Supervised Skeleton Representation Learningcited by 0
  16. 2026Diverse Yet Consistent: Context-Guided Diffusion with Energy-Based Joint Refinement for Multi-Agent Motion Predictioncited by 0
  17. 2026Deformable Gaussian Occupancy: Decoupling Rigid and Nonrigid Motion with Factorized Distillationcited by 0
  18. 2026Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-centric Videoscited by 0
  19. 2026Three-Step Hierarchical Transformer for Multi-Pedestrian Trajectory Predictioncited by 0
  20. 2026HumAIN: Human-Aware Implicit Social Robot Navigationcited by 0
  21. 2026EgoNeMo: Transferable Map of Pedestrian Dynamics via Egocentric LiDAR Scancited by 0