mlsanityData to 14 Oct 2026

Three Steps to Multimodal Trajectory Prediction: Modality Clustering, Classification and Synthesis

arXiv 2103.07854 ↗First posted 14 Mar 2021cs.CVcited by 46

Not on the map. Why?

Abstract (arXiv)

Multimodal prediction results are essential for trajectory prediction task as there is no single correct answer for the future. Previous frameworks can be divided into three categories: regression, generation and classification frameworks. However, these frameworks have weaknesses in different aspects so that they cannot model the multimodal prediction task comprehensively. In this paper, we present a novel insight along with a brand-new prediction framework by formulating multimodal prediction into three steps: modality clustering, classification and synthesis, and address the shortcomings of earlier frameworks. Exhaustive experiments on popular benchmarks have demonstrated that our proposed method surpasses state-of-the-art works even without introducing social and map information. Specifically, we achieve 19.2% and 20.8% improvement on ADE and FDE respectively on ETH/UCY dataset. Our code will be made publicly availabe.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 14 Mar 2021.

Builds on

24 of 39 references.

References · 24

  1. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  2. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 690
  3. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 420
  4. 2016RMPE: Regional Multi-person Pose Estimationcited by 340
  5. 2020Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Predictioncited by 325
  6. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  7. 2018SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraintscited by 298
  8. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  9. 2020It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Predictioncited by 236
  10. 2020Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Predictioncited by 218
  11. 2018CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmarkcited by 214
  12. 2017Social Attention: Modeling Attention in Human Crowdscited by 186
12 more
  1. 2019Peeking into the Future: Predicting Future Person Activities and Locations in Videoscited by 114
  2. 2018TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agentscited by 111
  3. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  4. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 80
  5. 2020TPNet: Trajectory Proposal Network for Motion Predictioncited by 80
  6. 2017Tracking The Untrackable: Learning To Track Multiple Cues with Long-Term Dependenciescited by 76
  7. 2019The Garden of Forking Paths: Towards Multi-Future Trajectory Predictioncited by 72
  8. 2020TubeTK: Adopting Tubes to Track Multi-Object in a One-Step Training Modelcited by 70
  9. 2020Recursive Social Behavior Graph for Trajectory Predictioncited by 66
  10. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  11. 2019Looking to Relations for Future Trajectory Forecastcited by 23
  12. 2018Survey on Vision-based Path Predictioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 11; 2023: 15; 2024: 10; 2025: 6; 2026: 2.

46 papers.

Sort:

By area:

  1. 2022Stochastic Trajectory Prediction via Motion Indeterminacy Diffusioncited by 185
  2. 2021Multi-Person 3D Motion Prediction with Multi-Range Transformerscited by 63
  3. 2022Adaptive Trajectory Prediction via Transferable GNNcited by 51
  4. 2022Learning Pedestrian Group Representations for Multi-modal Trajectory Predictioncited by 44
  5. 2023EigenTrajectory: Low-Rank Descriptors for Multi-Modal Trajectory Forecastingcited by 41
  6. 2024SingularTrajectory: Universal Trajectory Predictor Using Diffusion Modelcited by 35
  7. 2023FEND: A Future Enhanced Distribution-Aware Contrastive Learning Framework for Long-tail Trajectory Predictioncited by 31
  8. 2024Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Predictioncited by 31
  9. 2023Stochastic Multi-Person 3D Motion Forecastingcited by 28
  10. 2023Fast Inference and Update of Probabilistic Density Estimation on Trajectory Predictioncited by 27
  11. 2022SMEMO: Social Memory for Trajectory Forecastingcited by 26
  12. 2022GATraj: A Graph- and Attention-based Multi-Agent Trajectory Prediction Modelcited by 26
34 more
  1. 2024T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memorycited by 20
  2. 2021MSN: Multi-Style Network for Trajectory Predictioncited by 19
  3. 2022LatentFormer: Multi-Agent Transformer-Based Interaction Modeling and Trajectory Predictioncited by 19
  4. 2024Progressive Pretext Task Learning for Human Trajectory Predictioncited by 16
  5. 2022Forecasting Human Trajectory from Scene Historycited by 14
  6. 2023Uncertainty-Aware Pedestrian Trajectory Prediction via Distributional Diffusioncited by 13
  7. 2023Combating Representation Learning Disparity with Geometric Harmonizationcited by 11
  8. 2024Intention-aware Denoising Diffusion Model for Trajectory Predictioncited by 11
  9. 2022Highlighting Object Category Immunity for the Generalization of Human-Object Interaction Detectioncited by 9
  10. 2023Social Motion Prediction with Cognitive Hierarchiescited by 9
  11. 2024Recurrent Aligned Network for Generalized Pedestrian Trajectory Predictioncited by 9
  12. 2025BP-SGCN: Behavioral Pseudo-Label Informed Sparse Graph Convolution Network for Pedestrian and Heterogeneous Trajectory Predictioncited by 9
  13. 2023Vision-based Multi-future Trajectory Prediction: A Surveycited by 8
  14. 2023TrajPAC: Towards Robustness Verification of Pedestrian Trajectory Prediction Modelscited by 8
  15. 2022Predicting Hate Intensity of Twitter Conversation Threadscited by 7
  16. 2025Continuous Locomotive Crowd Behavior Generationcited by 7
  17. 2024MGF: Mixed Gaussian Flow for Diverse Trajectory Predictioncited by 6
  18. 2024Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowdscited by 6
  19. 2023Adaptive Human Trajectory Prediction via Latent Corridorscited by 5
  20. 2022Estimator: An Effective and Scalable Framework for Transportation Mode Classification over Trajectoriescited by 4
  21. 2023Sparse Pedestrian Character Learning for Trajectory Predictioncited by 3
  22. 2024RealTraj: Towards Real-World Pedestrian Trajectory Forecastingcited by 3
  23. 2025Stochastic Trajectory Prediction under Unstructured Constraintscited by 3
  24. 2023Integrating spoken instructions into flight trajectory prediction to optimize automation in air traffic controlcited by 2
  25. 2025Towards Predicting Any Human Trajectory In Contextcited by 2
  26. 2025Intention-Aware Diffusion Model for Pedestrian Trajectory Predictioncited by 2
  27. 2022Learning a Pedestrian Social Behavior Dictionarycited by 1
  28. 2023Differentiable Outlier Detection Enable Robust Deep Multimodal Analysiscited by 1
  29. 2024PMM-Net: Single-stage Multi-agent Trajectory Prediction with Patching-based Embedding and Explicit Modal Modulationcited by 1
  30. 2025Diffusion^2: Dual Diffusion Model with Uncertainty-Aware Adaptive Noise for Momentary Trajectory Predictioncited by 1
  31. 2026AGMA: Adaptive Gaussian Mixture Anchors for Prior-Guided Multimodal Human Trajectory Forecastingcited by 1
  32. 2023Likely, Light, and Accurate Context-Free Clusters-based Trajectory Predictioncited by 0
  33. 2023Learning Socio-Temporal Graphs for Multi-Agent Trajectory Predictioncited by 0
  34. 2026Hyperbolic Hierarchical Clustering for Visual Representation Learningcited by 0