# Learning Cooperative Trajectory Representations for Motion Forecasting

arXiv: https://arxiv.org/abs/2311.00371 · HTML: https://mlsanity.com/paper/2311.00371/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 69 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](https://mlsanity.com/paper/2203.10638/) (2022)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication](https://mlsanity.com/paper/2109.07644/) (2021)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction](https://mlsanity.com/paper/2008.07519/) (2020)
- [DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection](https://mlsanity.com/paper/2204.05575/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps](https://mlsanity.com/paper/2209.12836/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers](https://mlsanity.com/paper/2207.02202/) (2022)
- [Multimodal Motion Prediction with Stacked Transformers](https://mlsanity.com/paper/2103.11624/) (2021)
- [The ApolloScape Open Dataset for Autonomous Driving and its Application](https://mlsanity.com/paper/1803.06184/) (2018)

## Built on by (most cited first, 21 of 21)

- [End-to-End Autonomous Driving through V2X Cooperation](https://mlsanity.com/paper/2404.00717/) (2024)
- [V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2412.01812/) (2024)
- [CMP: Cooperative Motion Prediction with Multi-Agent Communication](https://mlsanity.com/paper/2403.17916/) (2024)
- [Co-MTP: A Cooperative Trajectory Prediction Framework with Multi-Temporal Fusion for Autonomous Driving](https://mlsanity.com/paper/2502.16589/) (2025)
- [Drones Help Drones: A Collaborative Framework for Multi-Drone Object Trajectory Prediction and Beyond](https://mlsanity.com/paper/2405.14674/) (2024)
- [CoopTrack: Exploring End-to-End Learning for Efficient Cooperative Sequential Perception](https://mlsanity.com/paper/2507.19239/) (2025)
- [CRUISE: Cooperative Reconstruction and Editing in V2X Scenarios using Gaussian Splatting](https://mlsanity.com/paper/2507.18473/) (2025)
- [TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2508.04682/) (2025)
- [Collaborative Perception Datasets for Autonomous Driving: A Review](https://mlsanity.com/paper/2504.12696/) (2025)
- [DONUT: A Decoder-Only Model for Trajectory Prediction](https://mlsanity.com/paper/2506.06854/) (2025)
- [LiDAR-based End-to-end Temporal Perception for Vehicle-Infrastructure Cooperation](https://mlsanity.com/paper/2411.14927/) (2024)
- [CooperRisk: A Driving Risk Quantification Pipeline with Multi-Agent Cooperative Perception and Prediction](https://mlsanity.com/paper/2506.15868/) (2025)
- [CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluation](https://mlsanity.com/paper/2603.21104/) (2026)
- [Conformal Trajectory Prediction with Multi-View Data Integration in Cooperative Driving](https://mlsanity.com/paper/2408.00374/) (2024)
- [Knowledge-Informed Multi-Agent Trajectory Prediction at Signalized Intersections for Infrastructure-to-Everything](https://mlsanity.com/paper/2501.13461/) (2025)
- [SGV3D:Towards Scenario Generalization for Vision-based Roadside 3D Object Detection](https://mlsanity.com/paper/2401.16110/) (2024)
- [Great GATsBi: Hybrid, Multimodal, Trajectory Forecasting for Bicycles using Anticipation Mechanism](https://mlsanity.com/paper/2508.14523/) (2025)
- [CoPAD : Multi-source Trajectory Fusion and Cooperative Trajectory Prediction with Anchor-oriented Decoder in V2X Scenarios](https://mlsanity.com/paper/2509.15984/) (2025)
- [V2X-RECT: An Efficient V2X Trajectory Prediction Framework via Redundant Interaction Filtering and Tracking Error Correction](https://mlsanity.com/paper/2511.17941/) (2025)
- [Towards Metric-Agnostic Trajectory Forecasting](https://mlsanity.com/paper/2607.01133/) (2026)
- [VIPS: Vehicle-Infrastructure Cooperative Planning Benchmark via Pseudo-Simulation](https://mlsanity.com/paper/2609.02462/) (2026)

## Abstract (arXiv)

Motion forecasting is an essential task for autonomous driving, and utilizing information from infrastructure and other vehicles can enhance forecasting capabilities. Existing research mainly focuses on leveraging single-frame cooperative information to enhance the limited perception capability of the ego vehicle, while underutilizing the motion and interaction context of traffic participants observed from cooperative devices. In this paper, we propose a forecasting-oriented representation paradigm to utilize motion and interaction features from cooperative information. Specifically, we present V2X-Graph, a representative framework to achieve interpretable and end-to-end trajectory feature fusion for cooperative motion forecasting. V2X-Graph is evaluated on V2X-Seq in vehicle-to-infrastructure (V2I) scenarios. To further evaluate on vehicle-to-everything (V2X) scenario, we construct the first real-world V2X motion forecasting dataset V2X-Traj, which contains multiple autonomous vehicles and infrastructure in every scenario. Experimental results on both V2X-Seq and V2X-Traj show the advantage of our method. We hope both V2X-Graph and V2X-Traj will benefit the further development of cooperative motion forecasting. Find the project at https://github.com/AIR-THU/V2X-Graph.