# MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying

arXiv: https://arxiv.org/abs/2306.17770 · HTML: https://mlsanity.com/paper/2306.17770/

## Key facts

- Cited by 140 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 74 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [CoverNet: Multimodal Behavior Prediction using Trajectory Sets](https://mlsanity.com/paper/1911.10298/) (2019)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings](https://mlsanity.com/paper/1905.01296/) (2019)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [Stratified Transformer for 3D Point Cloud Segmentation](https://mlsanity.com/paper/2203.14508/) (2022)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)

## Built on by (most cited first, 30 of 140)

- [AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.13757/) (2025)
- [EMMA: End-to-End Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2410.23262/) (2024)
- [DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving](https://mlsanity.com/paper/2503.07656/) (2025)
- [OpenEMMA: Open-Source Multimodal Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2412.15208/) (2024)
- [DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mamba](https://mlsanity.com/paper/2408.03601/) (2024)
- [Trajeglish: Traffic Modeling as Next-Token Prediction](https://mlsanity.com/paper/2312.04535/) (2023)
- [BehaviorGPT: Smart Agent Simulation for Autonomous Driving with Next-Patch Prediction](https://mlsanity.com/paper/2405.17372/) (2024)
- [SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout](https://mlsanity.com/paper/2412.12129/) (2024)
- [SparseAD: Sparse Query-Centric Paradigm for Efficient End-to-End Autonomous Driving](https://mlsanity.com/paper/2404.06892/) (2024)
- [DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Driving](https://mlsanity.com/paper/2310.05885/) (2023)
- [Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning](https://mlsanity.com/paper/2410.05582/) (2024)
- [Solving Motion Planning Tasks with a Scalable Generative Model](https://mlsanity.com/paper/2407.02797/) (2024)
- [Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Models](https://mlsanity.com/paper/2412.05334/) (2024)
- [WcDT: World-centric Diffusion Transformer for Traffic Scene Generation](https://mlsanity.com/paper/2404.02082/) (2024)
- [V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2412.01812/) (2024)
- [DriveGPT: Scaling Autoregressive Behavior Models for Driving](https://mlsanity.com/paper/2412.14415/) (2024)
- [SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Driving](https://mlsanity.com/paper/2402.02519/) (2024)
- [Improving Agent Behaviors with RL Fine-tuning for Autonomous Driving](https://mlsanity.com/paper/2409.18343/) (2024)
- [CMP: Cooperative Motion Prediction with Multi-Agent Communication](https://mlsanity.com/paper/2403.17916/) (2024)
- [SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction](https://mlsanity.com/paper/2405.15677/) (2024)
- [Open-sourced Data Ecosystem in Autonomous Driving: the Present and Future](https://mlsanity.com/paper/2312.03408/) (2023)
- [GuideFlow: Constraint-Guided Flow Matching for Planning in End-to-End Autonomous Driving](https://mlsanity.com/paper/2511.18729/) (2025)
- [Promptable Closed-loop Traffic Simulation](https://mlsanity.com/paper/2409.05863/) (2024)
- [Scaling Laws of Motion Forecasting and Planning -- Technical Report](https://mlsanity.com/paper/2506.08228/) (2025)
- [Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation](https://mlsanity.com/paper/2408.00766/) (2024)
- [GaussRender: Learning 3D Occupancy with Gaussian Rendering](https://mlsanity.com/paper/2502.05040/) (2025)
- [Trajectory Mamba: Efficient Attention-Mamba Forecasting Model Based on Selective SSM](https://mlsanity.com/paper/2503.10898/) (2025)
- [EDA: Evolving and Distinct Anchors for Multimodal Motion Prediction](https://mlsanity.com/paper/2312.09501/) (2023)
- [RoadPainter: Points Are Ideal Navigators for Topology transformER](https://mlsanity.com/paper/2407.15349/) (2024)
- [DriveAction: A Benchmark for Exploring Human-like Driving Decisions in VLA Models](https://mlsanity.com/paper/2506.05667/) (2025)

## Abstract (arXiv)

Motion prediction is crucial for autonomous driving systems to understand complex driving scenarios and make informed decisions. However, this task is challenging due to the diverse behaviors of traffic participants and complex environmental contexts. In this paper, we propose Motion TRansformer (MTR) frameworks to address these challenges. The initial MTR framework utilizes a transformer encoder-decoder structure with learnable intention queries, enabling efficient and accurate prediction of future trajectories. By customizing intention queries for distinct motion modalities, MTR improves multimodal motion prediction while reducing reliance on dense goal candidates. The framework comprises two essential processes: global intention localization, identifying the agent's intent to enhance overall efficiency, and local movement refinement, adaptively refining predicted trajectories for improved accuracy. Moreover, we introduce an advanced MTR++ framework, extending the capability of MTR to simultaneously predict multimodal motion for multiple agents. MTR++ incorporates symmetric context modeling and mutually-guided intention querying modules to facilitate future behavior interaction among multiple agents, resulting in scene-compliant future trajectories. Extensive experimental results demonstrate that the MTR framework achieves state-of-the-art performance on the highly-competitive motion prediction benchmarks, while the MTR++ framework surpasses its precursor, exhibiting enhanced performance and efficiency in predicting accurate multimodal future trajectories for multiple agents.