# MGTR: Multi-Granular Transformer for Motion Prediction with LiDAR

arXiv: https://arxiv.org/abs/2312.02409 · HTML: https://mlsanity.com/paper/2312.02409/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 39 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [MP3: A Unified Model to Map, Perceive, Predict and Plan](https://mlsanity.com/paper/2101.06806/) (2021)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying](https://mlsanity.com/paper/2306.17770/) (2023)
- [Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data](https://mlsanity.com/paper/1910.08233/) (2019)
- [MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps](https://mlsanity.com/paper/2003.06754/) (2020)
- [Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving](https://mlsanity.com/paper/1808.05819/) (2018)
- [TPNet: Trajectory Proposal Network for Motion Prediction](https://mlsanity.com/paper/2004.12255/) (2020)
- [LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perception](https://mlsanity.com/paper/2209.09385/) (2022)
- [MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction](https://mlsanity.com/paper/2006.03340/) (2020)
- [Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding](https://mlsanity.com/paper/2003.03212/) (2020)
- [MultiXNet: Multiclass Multistage Multimodal Motion Prediction](https://mlsanity.com/paper/2006.02000/) (2020)

## Built on by (most cited first, 9 of 9)

- [DriveGPT: Scaling Autoregressive Behavior Models for Driving](https://mlsanity.com/paper/2412.14415/) (2024)
- [Large Language Models Powered Context-aware Motion Prediction in Autonomous Driving](https://mlsanity.com/paper/2403.11057/) (2024)
- [Transfer Learning Study of Motion Transformer-based Trajectory Predictions](https://mlsanity.com/paper/2404.08271/) (2024)
- [AMEND: A Mixture of Experts Framework for Long-tailed Trajectory Prediction](https://mlsanity.com/paper/2402.08698/) (2024)
- [DuoSpaceNet: Leveraging Both Bird's-Eye-View and Perspective View Representations for 3D Object Detection](https://mlsanity.com/paper/2405.10577/) (2024)
- [Streaming Real-Time Trajectory Prediction Using Endpoint-Aware Modeling](https://mlsanity.com/paper/2603.01864/) (2026)
- [The Research of Group Re-identification from Multiple Cameras](https://mlsanity.com/paper/2407.14620/) (2024)
- [LiMTR: Time Series Motion Prediction for Diverse Road Users through Multimodal Feature Integration](https://mlsanity.com/paper/2410.15819/) (2024)
- [FlowS: One-Step Motion Prediction via Local Transport Conditioning](https://mlsanity.com/paper/2604.26065/) (2026)

## Abstract (arXiv)

Motion prediction has been an essential component of autonomous driving systems since it handles highly uncertain and complex scenarios involving moving agents of different types. In this paper, we propose a Multi-Granular TRansformer (MGTR) framework, an encoder-decoder network that exploits context features in different granularities for different kinds of traffic agents. To further enhance MGTR's capabilities, we leverage LiDAR point cloud data by incorporating LiDAR semantic features from an off-the-shelf LiDAR feature extractor. We evaluate MGTR on Waymo Open Dataset motion prediction benchmark and show that the proposed method achieved state-of-the-art performance, ranking 1st on its leaderboard (https://waymo.com/open/challenges/2023/motion-prediction/).