# LiMTR: Time Series Motion Prediction for Diverse Road Users through Multimodal Feature Integration

arXiv: https://arxiv.org/abs/2410.15819 · HTML: https://mlsanity.com/paper/2410.15819/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 26 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying](https://mlsanity.com/paper/2306.17770/) (2023)
- [SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/2210.07372/) (2022)
- [LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perception](https://mlsanity.com/paper/2209.09385/) (2022)
- [Optimal Linear Decay Learning Rate Schedules and Further Refinements](https://mlsanity.com/paper/2310.07831/) (2023)
- [MultiXNet: Multiclass Multistage Multimodal Motion Prediction](https://mlsanity.com/paper/2006.02000/) (2020)
- [WOMD-LiDAR: Raw Sensor Dataset Benchmark for Motion Forecasting](https://mlsanity.com/paper/2304.03834/) (2023)
- [Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints](https://mlsanity.com/paper/2306.01075/) (2023)
- [MGTR: Multi-Granular Transformer for Motion Prediction with LiDAR](https://mlsanity.com/paper/2312.02409/) (2023)
- [Robustness Benchmark of Road User Trajectory Prediction Models for Automated Driving](https://mlsanity.com/paper/2304.01895/) (2023)

## Abstract (arXiv)

Predicting the behavior of road users accurately is crucial to enable the safe operation of autonomous vehicles in urban or densely populated areas. Therefore, there has been a growing interest in time series motion prediction research, leading to significant advancements in state-of-the-art techniques in recent years. However, the potential of using LiDAR data to capture more detailed local features, such as a person's gaze or posture, remains largely unexplored. To address this, we develop a novel multimodal approach for motion prediction based on the PointNet foundation model architecture, incorporating local LiDAR features. Evaluation on the Waymo Open Dataset shows a performance improvement of 6.20% and 1.58% in minADE and mAP respectively, when integrated and compared with the previous state-of-the-art MTR. We open-source the code of our LiMTR model.