mlsanityData to 7 Oct 2026

Multi-Transmotion: Pre-trained Model for Human Motion Prediction

arXiv 2411.02673 ↗First posted 4 Nov 2024cs.CV cs.ROcited by 21

Not on the map. Why?

Abstract (arXiv)

The ability of intelligent systems to predict human behaviors is crucial, particularly in fields such as autonomous vehicle navigation and social robotics. However, the complexity of human motion have prevented the development of a standardized dataset for human motion prediction, thereby hindering the establishment of pre-trained models. In this paper, we address these limitations by integrating multiple datasets, encompassing both trajectory and 3D pose keypoints, to propose a pre-trained model for human motion prediction. We merge seven distinct datasets across varying modalities and standardize their formats. To facilitate multimodal pre-training, we introduce Multi-Transmotion, an innovative transformer-based model designed for cross-modality pre-training. Additionally, we present a novel masking strategy to capture rich representations. Our methodology demonstrates competitive performance across various datasets on several downstream tasks, including trajectory prediction in the NBA and JTA datasets, as well as pose prediction in the AMASS and 3DPW datasets. The code is publicly available: https://github.com/vita-epfl/multi-transmotion

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 53 of its 68 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

53 of 68 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Segment Anythingcited by 8,158
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  11. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
  12. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
38 more
  1. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 327
  2. 2020Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Predictioncited by 324
  3. 2018SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraintscited by 296
  4. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  5. 2019Learning Trajectory Dependencies for Human Motion Predictioncited by 245
  6. 2019Human Motion Trajectory Prediction: A Surveycited by 231
  7. 2019Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networkscited by 228
  8. 2018Convolutional Social Pooling for Vehicle Trajectory Predictioncited by 221
  9. 2022Stochastic Trajectory Prediction via Motion Indeterminacy Diffusioncited by 184
  10. 2020Transformer Networks for Trajectory Forecastingcited by 180
  11. 2020History Repeats Itself: Human Motion Prediction via Motion Attentioncited by 164
  12. 2018Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learningcited by 162
  13. 2021Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Predictioncited by 131
  14. 2023Leapfrog Diffusion Model for Stochastic Trajectory Predictioncited by 129
  15. 2020Human Trajectory Forecasting in Crowds: A Deep Learning Perspectivecited by 128
  16. 2022GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoningcited by 128
  17. 20234M: Massively Multimodal Masked Modelingcited by 118
  18. 2018Convolutional Sequence to Sequence Model for Human Dynamicscited by 115
  19. 2023EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoningcited by 97
  20. 2021Space-Time-Separable Graph Convolutional Network for Pose Forecastingcited by 84
  21. 2018Learning to Detect and Track Visible and Occluded Body Joints in a Virtual Worldcited by 81
  22. 2022Progressively Generating Better Initial Guesses Towards Next Stages for High-Quality Human Motion Predictioncited by 66
  23. 2018Action-Agnostic Human Pose Forecastingcited by 56
  24. 2020Collaborative Motion Prediction via Neural Motion Message Passingcited by 55
  25. 2021OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Associationcited by 55
  26. 2017Multimodal Probabilistic Model-Based Planning for Human-Robot Interactioncited by 50
  27. 2022Spatio-Temporal Gating-Adjacency GCN for Human Motion Predictioncited by 48
  28. 2020Multimodal Deep Generative Models for Trajectory Prediction: A Conditional Variational Autoencoder Approachcited by 47
  29. 2022Non-Probability Sampling Network for Stochastic Human Trajectory Predictioncited by 46
  30. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  31. 2024SingularTrajectory: Universal Trajectory Predictor Using Diffusion Modelcited by 35
  32. 2022A generic diffusion-based approach for 3D human pose prediction in the wildcited by 33
  33. 2023Social-Transmotion: Promptable Human Trajectory Predictioncited by 33
  34. 2024Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Predictioncited by 31
  35. 2020DAG-Net: Double Attentive Graph Neural Network for Trajectory Forecastingcited by 30
  36. 2023trajdata: A Unified Interface to Multiple Human Trajectory Datasetscited by 25
  37. 2022BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimationcited by 13
  38. 2023Toward Reliable Human Pose Forecasting with Uncertaintycited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 7.

21 papers.

Sort:

By area:

  1. 2025KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skillscited by 39
  2. 2025Multi-modal Knowledge Distillation-based Human Trajectory Forecastingcited by 10
  3. 2024MotionWavelet: Human Motion Prediction via Wavelet Manifold Learningcited by 8
  4. 2025Continuous Locomotive Crowd Behavior Generationcited by 7
  5. 2025Social-Pose: Enhancing Trajectory Prediction with Human Body Posecited by 5
  6. 2025JointDiff: Bridging Continuous and Discrete in Multi-Agent Trajectory Generationcited by 5
  7. 2024A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motionscited by 4
  8. 2025Unified Human Localization and Trajectory Prediction with Monocular Visioncited by 4
  9. 2025HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRAcited by 4
  10. 2026Heteroscedastic Diffusion for Multi-Agent Trajectory Modelingcited by 3
  11. 2024RealTraj: Towards Real-World Pedestrian Trajectory Forecastingcited by 2
  12. 2025Towards Predicting Any Human Trajectory In Contextcited by 2
9 more
  1. 2025HUMOF: Human Motion Forecasting in Interactive Social Scenescited by 2
  2. 2025Ratatouille: Imitation Learning Ingredients for Real-world Social Robot Navigationcited by 2
  3. 2026Social-Mamba: Socially-Aware Trajectory Forecasting with State-Space Modelscited by 1
  4. 2025Valeo Near-Field: a novel dataset for pedestrian intent detectioncited by 0
  5. 2026Robust Human Trajectory Prediction via Self-Supervised Skeleton Representation Learningcited by 0
  6. 2026SimpliHuMoN: Simplifying Human Motion Predictioncited by 0
  7. 2026Forecasting the Past: Gradient-Based Distribution Shift Detection in Trajectory Predictioncited by 0
  8. 2026EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecastingcited by 0
  9. 2026Deformable Gaussian Occupancy: Decoupling Rigid and Nonrigid Motion with Factorized Distillationcited by 0