# Dynamic Scenario Representation Learning for Motion Forecasting with Heterogeneous Graph Convolutional Recurrent Networks

arXiv: https://arxiv.org/abs/2303.04364 · HTML: https://mlsanity.com/paper/2303.04364/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 36 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks](https://mlsanity.com/paper/1809.10732/) (2018)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [Graph Neural Networks with convolutional ARMA filters](https://mlsanity.com/paper/1901.01343/) (2019)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [GOHOME: Graph-Oriented Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2109.01827/) (2021)
- [Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction](https://mlsanity.com/paper/2104.00563/) (2021)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting](https://mlsanity.com/paper/2101.06653/) (2021)
- [THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling](https://mlsanity.com/paper/2110.06607/) (2021)
- [TPCN: Temporal Point Cloud Networks for Motion Forecasting](https://mlsanity.com/paper/2103.03067/) (2021)
- [Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting](https://mlsanity.com/paper/1910.03650/) (2019)
- [What-If Motion Prediction for Autonomous Driving](https://mlsanity.com/paper/2008.10587/) (2020)
- [Learning to Predict Vehicle Trajectories with Model-based Planning](https://mlsanity.com/paper/2103.04027/) (2021)
- [Bootstrap Motion Forecasting With Self-Consistent Constraints](https://mlsanity.com/paper/2204.05859/) (2022)

## Built on by (most cited first, 28 of 28)

- [HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attention](https://mlsanity.com/paper/2404.06351/) (2024)
- [Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding](https://mlsanity.com/paper/2310.12970/) (2023)
- [SEPT: Towards Efficient Scene Representation Learning for Motion Prediction](https://mlsanity.com/paper/2309.15289/) (2023)
- [SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Driving](https://mlsanity.com/paper/2402.02519/) (2024)
- [SceneDM: Scene-level Multi-agent Trajectory Generation with Consistent Diffusion Models](https://mlsanity.com/paper/2311.15736/) (2023)
- [Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation](https://mlsanity.com/paper/2408.00766/) (2024)
- [Diffusion-Based Environment-Aware Trajectory Prediction](https://mlsanity.com/paper/2403.11643/) (2024)
- [Motion Forecasting in Continuous Driving](https://mlsanity.com/paper/2410.06007/) (2024)
- [Efficient Motion Prediction: A Lightweight & Accurate Trajectory Prediction Model With Fast Training and Inference Speed](https://mlsanity.com/paper/2409.16154/) (2024)
- [DONUT: A Decoder-Only Model for Trajectory Prediction](https://mlsanity.com/paper/2506.06854/) (2025)
- [GoIRL: Graph-Oriented Inverse Reinforcement Learning for Multimodal Trajectory Prediction](https://mlsanity.com/paper/2506.21121/) (2025)
- [FutureNet-LOF: Joint Trajectory Prediction and Lane Occupancy Field Prediction with Future Context Encoding](https://mlsanity.com/paper/2406.14422/) (2024)
- [Multi-Agent Trajectory Prediction with Difficulty-Guided Feature Enhancement Network](https://mlsanity.com/paper/2407.18551/) (2024)
- [Future-Aware Interaction Network For Motion Forecasting](https://mlsanity.com/paper/2503.06565/) (2025)
- [Controllable Diverse Sampling for Diffusion Based Motion Behavior Forecasting](https://mlsanity.com/paper/2402.03981/) (2024)
- [ProIn: Learning to Predict Trajectory Based on Progressive Interactions for Autonomous Driving](https://mlsanity.com/paper/2403.16374/) (2024)
- [Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models](https://mlsanity.com/paper/2407.19564/) (2024)
- [C-MAG: Cascade Multimodal Attributed Graphs for Supply Chain Link Prediction](https://mlsanity.com/paper/2508.08071/) (2025)
- [A Structure-Aware Lane Graph Transformer Model for Vehicle Trajectory Prediction](https://mlsanity.com/paper/2405.20121/) (2024)
- [HAMF: A Hybrid Attention-Mamba Framework for Joint Scene Context Understanding and Future Motion Representation Learning](https://mlsanity.com/paper/2505.15703/) (2025)
- [LANet: A Lane Boundaries-Aware Approach For Robust Trajectory Prediction](https://mlsanity.com/paper/2507.01308/) (2025)
- [HiMAP: History-aware Map-occupancy Prediction with Fallback](https://mlsanity.com/paper/2602.17231/) (2026)
- [Vectorized Representation Dreamer (VRD): Dreaming-Assisted Multi-Agent Motion-Forecasting](https://mlsanity.com/paper/2406.14415/) (2024)
- [Unifying Structural Proximity and Equivalence for Enhanced Dynamic Network Embedding](https://mlsanity.com/paper/2503.19926/) (2025)
- [ECTraj: Enhanced Consistency Training for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2605.08572/) (2026)
- [Unveiling Transferability in Trajectory Prediction via Latent Scene Embeddings](https://mlsanity.com/paper/2606.30777/) (2026)
- [A Unified Framework for Trajectory Prediction with Explicit Planning and Reaction Decomposition](https://mlsanity.com/paper/2608.05673/) (2026)
- [MoCAR: Motion-code Coordinate-aware AutoRegression for Continuous Trajectory Forecasting](https://mlsanity.com/paper/2610.06210/) (2026)

## Abstract (arXiv)

Due to the complex and changing interactions in dynamic scenarios, motion forecasting is a challenging problem in autonomous driving. Most existing works exploit static road graphs to characterize scenarios and are limited in modeling evolving spatio-temporal dependencies in dynamic scenarios. In this paper, we resort to dynamic heterogeneous graphs to model the scenario. Various scenario components including vehicles (agents) and lanes, multi-type interactions, and their changes over time are jointly encoded. Furthermore, we design a novel heterogeneous graph convolutional recurrent network, aggregating diverse interaction information and capturing their evolution, to learn to exploit intrinsic spatio-temporal dependencies in dynamic graphs and obtain effective representations of dynamic scenarios. Finally, with a motion forecasting decoder, our model predicts realistic and multi-modal future trajectories of agents and outperforms state-of-the-art published works on several motion forecasting benchmarks.