mlsanityData to 7 Oct 2026

Multimodal Motion Prediction with Stacked Transformers

arXiv 2103.11624 ↗First posted 22 Mar 2021cs.CV cs.AIcited by 207

Not on the map. Why?

Cited by 35 in End-to-end autonomous driving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Predicting multiple plausible future trajectories of the nearby vehicles is crucial for the safety of autonomous driving. Recent motion prediction approaches attempt to achieve such multimodal motion prediction by implicitly regularizing the feature or explicitly generating multiple candidate proposals. However, it remains challenging since the latent features may concentrate on the most frequent mode of the data while the proposal-based methods depend largely on the prior knowledge to generate and select the proposals. In this work, we propose a novel transformer framework for multimodal motion prediction, termed as mmTransformer. A novel network architecture based on stacked transformers is designed to model the multimodality at feature level with a set of fixed independent proposals. A region-based training strategy is then developed to induce the multimodality of the generated proposals. Experiments on Argoverse dataset show that the proposed model achieves the state-of-the-art performance on motion prediction, substantially improving the diversity and the accuracy of the predicted trajectories. Demo video and code are available at https://decisionforce.github.io/mmTransformer.

Key facts. Cited by 207 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 22 Mar 2021.

Builds on

28 of 36 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  5. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  6. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
  7. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  8. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  9. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 380
  10. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  11. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  12. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
16 more
  1. 2018SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraintscited by 296
  2. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  3. 2018Convolutional Social Pooling for Vehicle Trajectory Predictioncited by 221
  4. 2020Transformer Networks for Trajectory Forecastingcited by 180
  5. 2019Multiple Futures Predictioncited by 178
  6. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  7. 2018Auxiliary Tasks in Multi-task Learningcited by 106
  8. 2019Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecastingcited by 87
  9. 2018An LSTM Network for Highway Trajectory Predictioncited by 81
  10. 2020TPNet: Trajectory Proposal Network for Motion Predictioncited by 80
  11. 2020What-If Motion Prediction for Autonomous Drivingcited by 80
  12. 2019RWTH ASR Systems for LibriSpeech: Hybrid vs Attention -- w/o Data Augmentationcited by 63
  13. 2017Future Person Localization in First-Person Videoscited by 57
  14. 2020End-to-end Contextual Perception and Prediction with Interaction Transformercited by 54
  15. 2019Neural Turtle Graphics for Modeling City Road Layoutscited by 29
  16. 2019Lane Attention: Predicting Vehicles' Moving Trajectories by Learning Their Attention over Lanescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 48; 2023: 53; 2024: 46; 2025: 34; 2026: 13.

207 papers.

Sort:

By area:

  1. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  2. 2022Planning-oriented Autonomous Drivingcited by 1,068
  3. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  4. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  5. 2022Multimodal Learning with Transformers: A Surveycited by 399
  6. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  7. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  8. 2022Motion Transformer with Global Intention Localization and Local Movement Refinementcited by 267
  9. 2022VectorMapNet: End-to-end Vectorized HD Map Learningcited by 243
  10. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  11. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  12. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
38 more
  1. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  2. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 139
  3. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
  4. 2021HOME: Heatmap Output for future Motion Estimationcited by 122
  5. 2022PlanT: Explainable Planning Transformers via Object-Level Representationscited by 105
  6. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  7. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 100
  8. 2024DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mambacited by 87
  9. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  10. 2023Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoderscited by 72
  11. 2025ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Drivingcited by 66
  12. 2022ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planningcited by 63
  13. 2022Motion Inspired Unsupervised Perception and Prediction in Autonomous Drivingcited by 59
  14. 2023TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Predictioncited by 56
  15. 2023Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Trackingcited by 54
  16. 2023LAformer: Trajectory Prediction for Autonomous Driving with Lane-Aware Scene Constraintscited by 50
  17. 2022GANet: Goal Area Network for Motion Forecastingcited by 49
  18. 2024HIMap: HybrId Representation Learning for End-to-end Vectorized HD Map Constructioncited by 48
  19. 2023ProphNet: Efficient Agent-Centric Motion Forecasting with Anchor-Informed Proposalscited by 47
  20. 2023Leveraging Future Relationship Reasoning for Vehicle Trajectory Predictioncited by 47
  21. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  22. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Drivingcited by 43
  23. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  24. 2022FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphscited by 41
  25. 2022JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Drivingcited by 40
  26. 2023ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptationcited by 39
  27. 2023Frozen Transformers in Language Models Are Effective Visual Encoder Layerscited by 39
  28. 2021Container: Context Aggregation Networkcited by 37
  29. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  30. 2022Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrievalcited by 34
  31. 2023Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchorscited by 34
  32. 2023PPAD: Iterative Interactions of Prediction and Planning for End-to-end Autonomous Drivingcited by 33
  33. 2024SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Predictioncited by 33
  34. 2023MacFormer: Map-Agent Coupled Transformer for Real-time and Robust Trajectory Predictioncited by 32
  35. 2024Producing and Leveraging Online Map Uncertainty in Trajectory Predictioncited by 32
  36. 2022SSL-Lanes: Self-Supervised Learning for Motion Forecasting in Autonomous Drivingcited by 31
  37. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  38. 2022Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Predictioncited by 28

50 most cited of 207 All 207 (JSON)