# Relation-based Motion Prediction using Traffic Scene Graphs

arXiv: https://arxiv.org/abs/2212.02503 · HTML: https://mlsanity.com/paper/2212.02503/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 24 Nov 2022.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving](https://mlsanity.com/paper/2112.12610/) (2021)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [GRIP++: Enhanced Graph-based Interaction-aware Trajectory Prediction for Autonomous Driving](https://mlsanity.com/paper/1907.07792/) (2019)
- [Graph Neural Networks for Modelling Traffic Participant Interaction](https://mlsanity.com/paper/1903.01254/) (2019)
- [Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Tracking](https://mlsanity.com/paper/2102.09117/) (2021)
- [Deep Learning-based Vehicle Behaviour Prediction For Autonomous Driving Applications: A Review](https://mlsanity.com/paper/1912.11676/) (2019)
- [Graph and Recurrent Neural Network-based Vehicle Trajectory Prediction For Highway Driving](https://mlsanity.com/paper/2107.03663/) (2021)
- [A Review of Tracking, Prediction and Decision Making Methods for Autonomous Driving](https://mlsanity.com/paper/1909.07707/) (2019)
- [Towards Traffic Scene Description: The Semantic Scene Graph](https://mlsanity.com/paper/2111.10196/) (2021)
- [Situation-Aware Pedestrian Trajectory Prediction with Spatio-Temporal Attention Model](https://mlsanity.com/paper/1902.05437/) (2019)

## Built on by (most cited first, 6 of 6)

- [MTP-GO: Graph-Based Probabilistic Multi-Agent Trajectory Prediction with Neural ODEs](https://mlsanity.com/paper/2302.00735/) (2023)
- [SemanticFormer: Holistic and Semantic Traffic Scene Representation for Trajectory Prediction using Knowledge Graphs](https://mlsanity.com/paper/2404.19379/) (2024)
- [nuScenes Knowledge Graph -- A comprehensive semantic representation of traffic scenes for trajectory prediction](https://mlsanity.com/paper/2312.09676/) (2023)
- [SocialFormer: Social Interaction Modeling with Edge-enhanced Heterogeneous Graph Transformers for Trajectory Prediction](https://mlsanity.com/paper/2405.03809/) (2024)
- [DiVR: incorporating context from diverse VR scenes for human trajectory prediction](https://mlsanity.com/paper/2411.08409/) (2024)
- [Grounding Vision-Language Models in Driving Semantics: A Multi-Dataset Predicate Framework for Explainable Reasoning](https://mlsanity.com/paper/2609.31636/) (2026)

## Abstract (arXiv)

Representing relevant information of a traffic scene and understanding its environment is crucial for the success of autonomous driving. Modeling the surrounding of an autonomous car using semantic relations, i.e., how different traffic participants relate in the context of traffic rule based behaviors, is hardly been considered in previous work. This stems from the fact that these relations are hard to extract from real-world traffic scenes. In this work, we model traffic scenes in a form of spatial semantic scene graphs for various different predictions about the traffic participants, e.g., acceleration and deceleration. Our learning and inference approach uses Graph Neural Networks (GNNs) and shows that incorporating explicit information about the spatial semantic relations between traffic participants improves the predicdtion results. Specifically, the acceleration prediction of traffic participants is improved by up to 12% compared to the baselines, which do not exploit this explicit information. Furthermore, by including additional information about previous scenes, we achieve 73% improvements.