# ReCoG: A Deep Learning Framework with Heterogeneous Graph for Interaction-Aware Trajectory Prediction

arXiv: https://arxiv.org/abs/2012.05032 · HTML: https://mlsanity.com/paper/2012.05032/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 20 of 20)

- [GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2109.01827/) (2021)
- [HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding](https://mlsanity.com/paper/2205.09753/) (2022)
- [THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling](https://mlsanity.com/paper/2110.06607/) (2021)
- [Traj-MAE: Masked Autoencoders for Trajectory Prediction](https://mlsanity.com/paper/2303.06697/) (2023)
- [ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptation](https://mlsanity.com/paper/2307.14187/) (2023)
- [Imagining The Road Ahead: Multi-Agent Trajectory Prediction via Differentiable Simulation](https://mlsanity.com/paper/2104.11212/) (2021)
- [PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulation](https://mlsanity.com/paper/2109.11094/) (2021)
- [Graph Neural Networks in IoT: A Survey](https://mlsanity.com/paper/2203.15935/) (2022)
- [Machine Learning for Autonomous Vehicle's Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions](https://mlsanity.com/paper/2307.07527/) (2023)
- [A Diffusion-Model of Joint Interactive Navigation](https://mlsanity.com/paper/2309.12508/) (2023)
- [Holistic Graph-based Motion Prediction](https://mlsanity.com/paper/2301.13545/) (2023)
- [STrajNet: Multi-modal Hierarchical Transformer for Occupancy Flow Field Prediction in Autonomous Driving](https://mlsanity.com/paper/2208.00394/) (2022)
- [DiPA: Probabilistic Multi-Modal Interactive Prediction for Autonomous Driving](https://mlsanity.com/paper/2210.06106/) (2022)
- [SocialFormer: Social Interaction Modeling with Edge-enhanced Heterogeneous Graph Transformers for Trajectory Prediction](https://mlsanity.com/paper/2405.03809/) (2024)
- [MS-Net: A Multi-Path Sparse Model for Motion Prediction in Multi-Scenes](https://mlsanity.com/paper/2403.00353/) (2024)
- [Interpretable Goal-Based model for Vehicle Trajectory Prediction in Interactive Scenarios](https://mlsanity.com/paper/2308.04312/) (2023)
- [STF: Spatial Temporal Fusion for Trajectory Prediction](https://mlsanity.com/paper/2311.18149/) (2023)
- [Spatial-Temporal Conditional Random Field for Human Trajectory Prediction](https://mlsanity.com/paper/2311.18198/) (2023)
- [Jointly Learning Agent and Lane Information for Multimodal Trajectory Prediction](https://mlsanity.com/paper/2111.13350/) (2021)
- [ProgD: Progressive Multi-scale Decoding with Dynamic Graphs for Joint Multi-agent Motion Forecasting](https://mlsanity.com/paper/2509.09210/) (2025)

## Abstract (arXiv)

Predicting the future trajectory of surrounding vehicles is essential for the navigation of autonomous vehicles in complex real-world driving scenarios. It is challenging as a vehicle's motion is affected by many factors, including its surrounding infrastructures and vehicles. In this work, we develop the ReCoG (Recurrent Convolutional and Graph Neural Networks), which is a general scheme that represents vehicle interactions with infrastructure information as a heterogeneous graph and applies graph neural networks (GNNs) to model the high-level interactions for trajectory prediction. Nodes in the graph contain corresponding features, where a vehicle node contains its sequential feature encoded using Recurrent Neural Network (RNN), and an infrastructure node contains spatial feature encoded using Convolutional Neural Network (CNN). Then the ReCoG predicts the future trajectory of the target vehicle by jointly considering all of the features. Experiments are conducted by using the INTERACTION dataset. Experimental results show that the proposed ReCoG outperforms other state-of-the-art methods in terms of different types of displacement error, validating the feasibility and effectiveness of the developed approach.