# Co-MTP: A Cooperative Trajectory Prediction Framework with Multi-Temporal Fusion for Autonomous Driving

arXiv: https://arxiv.org/abs/2502.16589 · HTML: https://mlsanity.com/paper/2502.16589/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 23 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication](https://mlsanity.com/paper/2109.07644/) (2021)
- [V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction](https://mlsanity.com/paper/2008.07519/) (2020)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception](https://mlsanity.com/paper/2303.07601/) (2023)
- [MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying](https://mlsanity.com/paper/2306.17770/) (2023)
- [V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting](https://mlsanity.com/paper/2305.05938/) (2023)
- [LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting](https://mlsanity.com/paper/2101.06653/) (2021)
- [HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding](https://mlsanity.com/paper/2205.09753/) (2022)
- [PiP: Planning-informed Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/2003.11476/) (2020)
- [Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Driving](https://mlsanity.com/paper/2207.10422/) (2022)
- [V2X-Real: a Large-Scale Dataset for Vehicle-to-Everything Cooperative Perception](https://mlsanity.com/paper/2403.16034/) (2024)
- [Score refinement for confidence-based 3D multi-object tracking](https://mlsanity.com/paper/2107.04327/) (2021)
- [QCNeXt: A Next-Generation Framework For Joint Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2306.10508/) (2023)
- [V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2412.01812/) (2024)
- [Differentiable Gaussian Process Motion Planning](https://mlsanity.com/paper/1907.09591/) (2019)
- [Deep Interactive Motion Prediction and Planning: Playing Games with Motion Prediction Models](https://mlsanity.com/paper/2204.02392/) (2022)
- [CMP: Cooperative Motion Prediction with Multi-Agent Communication](https://mlsanity.com/paper/2403.17916/) (2024)
- [Learning Cooperative Trajectory Representations for Motion Forecasting](https://mlsanity.com/paper/2311.00371/) (2023)
- [Deep Structured Reactive Planning](https://mlsanity.com/paper/2101.06832/) (2021)
- [Analyzing Infrastructure LiDAR Placement with Realistic LiDAR Simulation Library](https://mlsanity.com/paper/2211.15975/) (2022)
- [Scene Informer: Anchor-based Occlusion Inference and Trajectory Prediction in Partially Observable Environments](https://mlsanity.com/paper/2309.13893/) (2023)
- [Interventional Behavior Prediction: Avoiding Overly Confident Anticipation in Interactive Prediction](https://mlsanity.com/paper/2204.08665/) (2022)

## Built on by (most cited first, 14 of 14)

- [CoopTrack: Exploring End-to-End Learning for Efficient Cooperative Sequential Perception](https://mlsanity.com/paper/2507.19239/) (2025)
- [QuantV2X: A Fully Quantized Multi-Agent System for Cooperative Perception](https://mlsanity.com/paper/2509.03704/) (2025)
- [TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2508.04682/) (2025)
- [M3CAD: Towards Generic Cooperative Autonomous Driving Benchmark](https://mlsanity.com/paper/2505.06746/) (2025)
- [GEMINUS: Dual-aware Global and Scene-Adaptive Mixture-of-Experts for End-to-End Autonomous Driving](https://mlsanity.com/paper/2507.14456/) (2025)
- [Risk Map As Middleware: Towards Interpretable Cooperative End-to-end Autonomous Driving for Risk-Aware Planning](https://mlsanity.com/paper/2508.07686/) (2025)
- [CooperRisk: A Driving Risk Quantification Pipeline with Multi-Agent Cooperative Perception and Prediction](https://mlsanity.com/paper/2506.15868/) (2025)
- [SEAL: Vision-Language Model-Based Safe End-to-End Cooperative Autonomous Driving with Adaptive Long-Tail Modeling](https://mlsanity.com/paper/2506.21041/) (2025)
- [Linking Modality Isolation in Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2603.00609/) (2026)
- [Background Fades, Foreground Leads: Curriculum-Guided Background Pruning for Efficient Foreground-Centric Collaborative Perception](https://mlsanity.com/paper/2510.19250/) (2025)
- [Driving in Corner Case: A Real-World Adversarial Closed-Loop Evaluation Platform for End-to-End Autonomous Driving](https://mlsanity.com/paper/2512.16055/) (2025)
- [V2X-RECT: An Efficient V2X Trajectory Prediction Framework via Redundant Interaction Filtering and Tracking Error Correction](https://mlsanity.com/paper/2511.17941/) (2025)
- [CooperDrive: Enhancing Driving Decisions Through Cooperative Perception](https://mlsanity.com/paper/2604.14454/) (2026)
- [VIPS: Vehicle-Infrastructure Cooperative Planning Benchmark via Pseudo-Simulation](https://mlsanity.com/paper/2609.02462/) (2026)

## Abstract (arXiv)

Vehicle-to-everything technologies (V2X) have become an ideal paradigm to extend the perception range and see through the occlusion. Exiting efforts focus on single-frame cooperative perception, however, how to capture the temporal cue between frames with V2X to facilitate the prediction task even the planning task is still underexplored. In this paper, we introduce the Co-MTP, a general cooperative trajectory prediction framework with multi-temporal fusion for autonomous driving, which leverages the V2X system to fully capture the interaction among agents in both history and future domains to benefit the planning. In the history domain, V2X can complement the incomplete history trajectory in single-vehicle perception, and we design a heterogeneous graph transformer to learn the fusion of the history feature from multiple agents and capture the history interaction. Moreover, the goal of prediction is to support future planning. Thus, in the future domain, V2X can provide the prediction results of surrounding objects, and we further extend the graph transformer to capture the future interaction among the ego planning and the other vehicles' intentions and obtain the final future scenario state under a certain planning action. We evaluate the Co-MTP framework on the real-world dataset V2X-Seq, and the results show that Co-MTP achieves state-of-the-art performance and that both history and future fusion can greatly benefit prediction.