mlsanityData to 7 Oct 2026

Motion Transformer with Global Intention Localization and Local Movement Refinement

arXiv 2209.13508 ↗First posted 27 Sep 2022cs.CVcited by 267

On the map: World models → World models for autonomous driving

Abstract (arXiv)

Predicting multimodal future behavior of traffic participants is essential for robotic vehicles to make safe decisions. Existing works explore to directly predict future trajectories based on latent features or utilize dense goal candidates to identify agent's destinations, where the former strategy converges slowly since all motion modes are derived from the same feature while the latter strategy has efficiency issue since its performance highly relies on the density of goal candidates. In this paper, we propose Motion TRansformer (MTR) framework that models motion prediction as the joint optimization of global intention localization and local movement refinement. Instead of using goal candidates, MTR incorporates spatial intention priors by adopting a small set of learnable motion query pairs. Each motion query pair takes charge of trajectory prediction and refinement for a specific motion mode, which stabilizes the training process and facilitates better multimodal predictions. Experiments show that MTR achieves state-of-the-art performance on both the marginal and joint motion prediction challenges, ranking 1st on the leaderboards of Waymo Open Motion Dataset. The source code is available at https://github.com/sshaoshuai/MTR.

Key facts. Cited by 267 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 54 of its 71 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

54 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Sets23 Nov

Other references · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2017Non-local Neural Networkscited by 2,290
  9. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  10. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
  11. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  12. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
38 more
  1. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  2. 2021Conditional DETR for Fast Training Convergencecited by 422
  3. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  4. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  5. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  6. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  7. 2021DenseTNT: End-to-end Trajectory Prediction from Dense Goal Setscited by 292
  8. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  9. 2020It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Predictioncited by 235
  10. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  11. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  12. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
  13. 2019Multiple Futures Predictioncited by 178
  14. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166
  15. 2021IntentNet: Learning to Predict Intention from Raw Sensor Datacited by 158
  16. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  17. 2021HOME: Heatmap Output for future Motion Estimationcited by 122
  18. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  19. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  20. 2022HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encodingcited by 109
  21. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  22. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
  23. 2021TPCN: Temporal Point Cloud Networks for Motion Forecastingcited by 94
  24. 2018Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Drivingcited by 90
  25. 2019Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecastingcited by 87
  26. 2022M2I: From Factored Marginal Trajectory Prediction to Interactive Predictioncited by 81
  27. 2020TPNet: Trajectory Proposal Network for Motion Predictioncited by 80
  28. 2020MANTRA: Memory Augmented Networks for Multiple Trajectory Predictioncited by 69
  29. 2020Diverse and Admissible Trajectory Forecasting through Multimodal Context Understandingcited by 68
  30. 2022MPPNet: Multi-Frame Feature Intertwining with Proxy Points for 3D Temporal Object Detectioncited by 65
  31. 2021Multi-Person 3D Motion Prediction with Multi-Range Transformerscited by 63
  32. 2021Identifying Driver Interactions via Conditional Behavior Predictioncited by 48
  33. 2022A Unified Query-based Paradigm for Point Cloud Understandingcited by 32
  34. 2020PRANK: motion Prediction based on RANKingcited by 22
  35. 2022MotionCNN: A Strong Baseline for Motion Prediction in Autonomous Drivingcited by 21
  36. 2019StarNet: Pedestrian Trajectory Prediction using Deep Neural Network in Star Topologycited by 17
  37. 2022MTR-A: 1st Place Solution for 2022 Waymo Open Dataset Challenge -- Motion Predictioncited by 17
  38. 2022TENET: Transformer Encoding Network for Effective Temporal Flow on Motion Predictioncited by 14

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 46; 2024: 74; 2025: 80; 2026: 63.

267 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2023LMDrive: Closed-Loop End-to-End Driving with Large Language Modelscited by 301
  3. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  4. 2023Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Drivingcited by 168
  5. 2023A Language Agent for Autonomous Drivingcited by 161
  6. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  7. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  8. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  9. 2023MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Queryingcited by 139
  10. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  11. 2024DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mambacited by 87
  12. 2023The Waymo Open Sim Agents Challengecited by 86
38 more
  1. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  2. 2023UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representationcited by 74
  3. 2023Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoderscited by 72
  4. 2025Robust Autonomy Emerges from Self-Playcited by 64
  5. 2023Trajeglish: Traffic Modeling as Next-Token Predictioncited by 59
  6. 2023Augmenting Lane Perception and Topology Understanding with Standard Definition Navigation Mapscited by 58
  7. 2023Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Trackingcited by 54
  8. 2023CAT: Closed-loop Adversarial Training for Safe End-to-End Drivingcited by 52
  9. 2024UniTraj: A Unified Framework for Scalable Vehicle Trajectory Predictioncited by 51
  10. 2024BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Predictioncited by 49
  11. 2023DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Drivingcited by 46
  12. 2023RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarioscited by 46
  13. 2024SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Drivingcited by 46
  14. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  15. 2022FJMP: Factorized Joint Multi-Agent Motion Prediction over Learned Directed Acyclic Interaction Graphscited by 41
  16. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  17. 2024Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Modelscited by 40
  18. 2024HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attentioncited by 37
  19. 2024LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Modelscited by 35
  20. 2024SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Predictioncited by 33
  21. 2023MacFormer: Map-Agent Coupled Transformer for Real-time and Robust Trajectory Predictioncited by 32
  22. 2023Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encodingcited by 32
  23. 2024WcDT: World-centric Diffusion Transformer for Traffic Scene Generationcited by 31
  24. 2025CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Drivingcited by 31
  25. 2025TA-VLA: Elucidating the Design Space of Torque-aware Vision-Language-Action Modelscited by 30
  26. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  27. 2024DriveGPT: Scaling Autoregressive Behavior Models for Drivingcited by 29
  28. 2023WOMD-LiDAR: Raw Sensor Dataset Benchmark for Motion Forecastingcited by 28
  29. 2023SEPT: Towards Efficient Scene Representation Learning for Motion Predictioncited by 27
  30. 2024SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Drivingcited by 27
  31. 2024Improving Agent Behaviors with RL Fine-tuning for Autonomous Drivingcited by 27
  32. 2023DriveSceneGen: Generating Diverse and Realistic Driving Scenarios from Scratchcited by 26
  33. 2023Multiverse Transformer: 1st Place Solution for Waymo Open Sim Agents Challenge 2023cited by 25
  34. 2024CMP: Cooperative Motion Prediction with Multi-Agent Communicationcited by 25
  35. 2024SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Predictioncited by 25
  36. 2023SceneDM: Scene-level Multi-agent Trajectory Generation with Consistent Diffusion Modelscited by 24
  37. 2023Large Trajectory Models are Scalable Motion Predictors and Plannerscited by 23
  38. 2024Generalizing Motion Planners with Mixture of Experts for Autonomous Drivingcited by 22

50 most cited of 267 All 267 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectories
  2. 2022PreTR: Spatio-Temporal Non-Autoregressive Trajectory Prediction Transformer
  3. 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives
  4. 2024DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States
  5. 2025GContextFormer: A global context-aware hybrid multi-head attention approach with scaled additive aggregation for multimodal trajectory prediction
  6. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networks
  7. 2025Intention-Aware Diffusion Model for Pedestrian Trajectory Prediction
  8. 2023Efficient Baselines for Motion Prediction in Autonomous Driving