# SCENE: Reasoning about Traffic Scenes using Heterogeneous Graph Neural Networks

arXiv: https://arxiv.org/abs/2301.03512 · HTML: https://mlsanity.com/paper/2301.03512/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 9 Jan 2023.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction](https://mlsanity.com/paper/1910.05449/) (2019)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [Residual Gated Graph ConvNets](https://mlsanity.com/paper/1711.07553/) (2017)
- [Universal Invariant and Equivariant Graph Neural Networks](https://mlsanity.com/paper/1905.04943/) (2019)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [IntentNet: Learning to Predict Intention from Raw Sensor Data](https://mlsanity.com/paper/2101.07907/) (2021)
- [HOME: Heatmap Output for future Motion Estimation](https://mlsanity.com/paper/2105.10968/) (2021)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting](https://mlsanity.com/paper/2101.06653/) (2021)
- [TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents](https://mlsanity.com/paper/1811.02146/) (2018)
- [Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving](https://mlsanity.com/paper/1808.05819/) (2018)
- [What-If Motion Prediction for Autonomous Driving](https://mlsanity.com/paper/2008.10587/) (2020)
- [Learning 3D-aware Egocentric Spatial-Temporal Interaction via Graph Convolutional Networks](https://mlsanity.com/paper/1909.09272/) (2019)
- [CRAT-Pred: Vehicle Trajectory Prediction with Crystal Graph Convolutional Neural Networks and Multi-Head Self-Attention](https://mlsanity.com/paper/2202.04488/) (2022)
- [Scenario-Transferable Semantic Graph Reasoning for Interaction-Aware Probabilistic Prediction](https://mlsanity.com/paper/2004.03053/) (2020)
- [Interaction-Based Trajectory Prediction Over a Hybrid Traffic Graph](https://mlsanity.com/paper/2009.12916/) (2020)
- [Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network](https://mlsanity.com/paper/2002.06241/) (2020)
- [Road Scene Graph: A Semantic Graph-Based Scene Representation Dataset for Intelligent Vehicles](https://mlsanity.com/paper/2011.13588/) (2020)

## Built on by (most cited first, 18 of 18)

- [SemanticFormer: Holistic and Semantic Traffic Scene Representation for Trajectory Prediction using Knowledge Graphs](https://mlsanity.com/paper/2404.19379/) (2024)
- [RedMotion: Motion Prediction via Redundancy Reduction](https://mlsanity.com/paper/2306.10840/) (2023)
- [Exploring Navigation Maps for Learning-Based Motion Prediction](https://mlsanity.com/paper/2302.06195/) (2023)
- [TempBEV: Improving Learned BEV Encoders with Combined Image and BEV Space Temporal Aggregation](https://mlsanity.com/paper/2404.11803/) (2024)
- [RESET: Revisiting Trajectory Sets for Conditional Behavior Prediction](https://mlsanity.com/paper/2304.05856/) (2023)
- [Exploring the Potential of Multi-Modal AI for Driving Hazard Prediction](https://mlsanity.com/paper/2310.04671/) (2023)
- [Heterogeneous Graph-based Trajectory Prediction using Local Map Context and Social Interactions](https://mlsanity.com/paper/2311.18553/) (2023)
- [On the Federated Learning Framework for Cooperative Perception](https://mlsanity.com/paper/2404.17147/) (2024)
- [DiffSemanticFusion: Semantic Raster BEV Fusion for Autonomous Driving via Online HD Map Diffusion](https://mlsanity.com/paper/2508.01778/) (2025)
- [LMR: Lane Distance-Based Metric for Trajectory Prediction](https://mlsanity.com/paper/2304.05869/) (2023)
- [Semantic Map Learning of Traffic Light to Lane Assignment based on Motion Data](https://mlsanity.com/paper/2309.14793/) (2023)
- [PreGSU-A Generalized Traffic Scene Understanding Model for Autonomous Driving based on Pre-trained Graph Attention Network](https://mlsanity.com/paper/2404.10263/) (2024)
- [Explainable Scene Understanding with Qualitative Representations and Graph Neural Networks](https://mlsanity.com/paper/2504.12817/) (2025)
- [GRLinQ: An Intelligent Spectrum Sharing Mechanism for Device-to-Device Communications with Graph Reinforcement Learning](https://mlsanity.com/paper/2408.09394/) (2024)
- [High-Order Evolving Graphs for Enhanced Representation of Traffic Dynamics](https://mlsanity.com/paper/2409.11206/) (2024)
- [GHR-VQA: Graph-guided Hierarchical Relational Reasoning for Video Question Answering](https://mlsanity.com/paper/2511.20201/) (2025)
- [Enhanced Feature Extraction for IoT Network Intrusion Detection Using GNNs and KAN](https://mlsanity.com/paper/2607.02981/) (2026)
- [Beamforming Design Via GNN in mmWave Cell-Free Massive MIMO Using Sub-6 GHz CSI](https://mlsanity.com/paper/2608.30524/) (2026)

## Abstract (arXiv)

Understanding traffic scenes requires considering heterogeneous information about dynamic agents and the static infrastructure. In this work we propose SCENE, a methodology to encode diverse traffic scenes in heterogeneous graphs and to reason about these graphs using a heterogeneous Graph Neural Network encoder and task-specific decoders. The heterogeneous graphs, whose structures are defined by an ontology, consist of different nodes with type-specific node features and different relations with type-specific edge features. In order to exploit all the information given by these graphs, we propose to use cascaded layers of graph convolution. The result is an encoding of the scene. Task-specific decoders can be applied to predict desired attributes of the scene. Extensive evaluation on two diverse binary node classification tasks show the main strength of this methodology: despite being generic, it even manages to outperform task-specific baselines. The further application of our methodology to the task of node classification in various knowledge graphs shows its transferability to other domains.