# MultiXNet: Multiclass Multistage Multimodal Motion Prediction

arXiv: https://arxiv.org/abs/2006.02000 · HTML: https://mlsanity.com/paper/2006.02000/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2020.

## Built on by (most cited first, 30 of 31)

- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planning](https://mlsanity.com/paper/2206.13387/) (2022)
- [StretchBEV: Stretching Future Instance Prediction Spatially and Temporally](https://mlsanity.com/paper/2203.13641/) (2022)
- [PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation](https://mlsanity.com/paper/2109.11094/) (2021)
- [Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Prediction](https://mlsanity.com/paper/2212.02181/) (2022)
- [Learning and Aggregating Lane Graphs for Urban Automated Driving](https://mlsanity.com/paper/2302.06175/) (2023)
- [Heterogeneous-Agent Trajectory Forecasting Incorporating Class Uncertainty](https://mlsanity.com/paper/2104.12446/) (2021)
- [MVFuseNet: Improving End-to-End Object Detection and Motion Forecasting through Multi-View Fusion of LiDAR Data](https://mlsanity.com/paper/2104.10772/) (2021)
- [DeTra: A Unified Model for Object Detection and Trajectory Forecasting](https://mlsanity.com/paper/2406.04426/) (2024)
- [StopNet: Scalable Trajectory and Occupancy Prediction for Urban Autonomous Driving](https://mlsanity.com/paper/2206.00991/) (2022)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints](https://mlsanity.com/paper/2306.01075/) (2023)
- [VT-Former: An Exploratory Study on Vehicle Trajectory Prediction for Highway Surveillance through Graph Isomorphism and Transformer](https://mlsanity.com/paper/2311.06623/) (2023)
- [Self-Supervised Action-Space Prediction for Automated Driving](https://mlsanity.com/paper/2109.10024/) (2021)
- [MGTR: Multi-Granular Transformer for Motion Prediction with LiDAR](https://mlsanity.com/paper/2312.02409/) (2023)
- [MoST: Multi-modality Scene Tokenization for Motion Prediction](https://mlsanity.com/paper/2404.19531/) (2024)
- [M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving](https://mlsanity.com/paper/2403.12552/) (2024)
- [Joint Perception and Prediction for Autonomous Driving: A Survey](https://mlsanity.com/paper/2412.14088/) (2024)
- [Practical Collaborative Perception: A Framework for Asynchronous and Multi-Agent 3D Object Detection](https://mlsanity.com/paper/2307.01462/) (2023)
- [Divide and Merge: Motion and Semantic Learning in End-to-End Autonomous Driving](https://mlsanity.com/paper/2502.07631/) (2025)
- [Investigating the Effect of Sensor Modalities in Multi-Sensor Detection-Prediction Models](https://mlsanity.com/paper/2101.03279/) (2021)
- [Convolutions for Spatial Interaction Modeling](https://mlsanity.com/paper/2104.07182/) (2021)
- [Aligning Bird-Eye View Representation of Point Cloud Sequences using Scene Flow](https://mlsanity.com/paper/2305.02909/) (2023)
- [LeTFuser: Light-weight End-to-end Transformer-Based Sensor Fusion for Autonomous Driving with Multi-Task Learning](https://mlsanity.com/paper/2310.13135/) (2023)
- [To New Beginnings: A Survey of Unified Perception in Autonomous Vehicle Software](https://mlsanity.com/paper/2508.20892/) (2025)
- [Motion Prediction Performance Analysis for Autonomous Driving Systems and the Effects of Tracking Noise](https://mlsanity.com/paper/2104.08368/) (2021)
- [Trajectory Prediction using Generative Adversarial Network in Multi-Class Scenarios](https://mlsanity.com/paper/2110.11401/) (2021)

## Abstract (arXiv)

One of the critical pieces of the self-driving puzzle is understanding the surroundings of a self-driving vehicle (SDV) and predicting how these surroundings will change in the near future. To address this task we propose MultiXNet, an end-to-end approach for detection and motion prediction based directly on lidar sensor data. This approach builds on prior work by handling multiple classes of traffic actors, adding a jointly trained second-stage trajectory refinement step, and producing a multimodal probability distribution over future actor motion that includes both multiple discrete traffic behaviors and calibrated continuous position uncertainties. The method was evaluated on large-scale, real-world data collected by a fleet of SDVs in several cities, with the results indicating that it outperforms existing state-of-the-art approaches.