# Spatio-Temporal Scene-Graph Embedding for Autonomous Vehicle Collision Prediction

arXiv: https://arxiv.org/abs/2111.06123 · HTML: https://mlsanity.com/paper/2111.06123/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 11 Nov 2021.

## Builds on (most cited first, 16 of 16)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Self-Attention Graph Pooling](https://mlsanity.com/paper/1904.08082/) (2019)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [On a Formal Model of Safe and Scalable Self-driving Cars](https://mlsanity.com/paper/1708.06374/) (2017)
- [Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning](https://mlsanity.com/paper/1811.02307/) (2018)
- [Scene-Graph Augmented Data-Driven Risk Assessment of Autonomous Vehicle Decisions](https://mlsanity.com/paper/2009.06435/) (2020)
- [When, Where, and What? A New Dataset for Anomaly Detection in Driving Videos](https://mlsanity.com/paper/2004.03044/) (2020)
- [Learning 3D-aware Egocentric Spatial-Temporal Interaction via Graph Convolutional Networks](https://mlsanity.com/paper/1909.09272/) (2019)
- [Deep Predictive Models for Collision Risk Assessment in Autonomous Driving](https://mlsanity.com/paper/1711.10453/) (2017)
- [Understanding Dynamic Scenes using Graph Convolution Networks](https://mlsanity.com/paper/2005.04437/) (2020)
- [Towards Accurate Vehicle Behaviour Classification With Multi-Relational Graph Convolutional Networks](https://mlsanity.com/paper/2002.00786/) (2020)
- [Opportunistic Intermittent Control with Safety Guarantees for Autonomous Systems](https://mlsanity.com/paper/2005.03726/) (2020)

## Built on by (most cited first, 29 of 29)

- [Graph-based Topology Reasoning for Driving Scenes](https://mlsanity.com/paper/2304.05277/) (2023)
- [Advancing Video Anomaly Detection: A Concise Review and a New Dataset](https://mlsanity.com/paper/2402.04857/) (2024)
- [Abductive Ego-View Accident Video Understanding for Safe Driving Perception](https://mlsanity.com/paper/2403.00436/) (2024)
- [Vision-Based Traffic Accident Detection and Anticipation: A Survey](https://mlsanity.com/paper/2308.15985/) (2023)
- [Behavioral Intention Prediction in Driving Scenes: A Survey](https://mlsanity.com/paper/2211.00385/) (2022)
- [Nexar Dashcam Collision Prediction Dataset and Challenge](https://mlsanity.com/paper/2503.03848/) (2025)
- [RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding](https://mlsanity.com/paper/2304.08600/) (2023)
- [Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark](https://mlsanity.com/paper/2212.09381/) (2022)
- [Spatial-Temporal-Aware Safe Multi-Agent Reinforcement Learning of Connected Autonomous Vehicles in Challenging Scenarios](https://mlsanity.com/paper/2210.02300/) (2022)
- [Survey on video anomaly detection in dynamic scenes with moving cameras](https://mlsanity.com/paper/2308.07050/) (2023)
- [Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding](https://mlsanity.com/paper/2407.05910/) (2024)
- [Explainable Action Prediction through Self-Supervision on Scene Graphs](https://mlsanity.com/paper/2302.03477/) (2023)
- [MaGNAS: A Mapping-Aware Graph Neural Architecture Search Framework for Heterogeneous MPSoC Deployment](https://mlsanity.com/paper/2307.08065/) (2023)
- [World Model-Based End-to-End Scene Generation for Accident Anticipation in Autonomous Driving](https://mlsanity.com/paper/2507.12762/) (2025)
- [GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving](https://mlsanity.com/paper/2511.11266/) (2025)
- [AI2-Active Safety: AI-enabled Interaction-aware Active Safety Analysis with Vehicle Dynamics](https://mlsanity.com/paper/2505.00322/) (2025)
- [BADAS: Context Aware Collision Prediction Using Real-World Dashcam Data](https://mlsanity.com/paper/2510.14876/) (2025)
- [Interpretable End-to-End Driving Model for Implicit Scene Understanding](https://mlsanity.com/paper/2308.01180/) (2023)
- [Deep Learning Advances in Vision-Based Traffic Accident Anticipation: A Comprehensive Review of Methods, Datasets, and Future Directions](https://mlsanity.com/paper/2505.07611/) (2025)
- [What Demands Attention in Urban Street Scenes? From Scene Understanding towards Road Safety: A Survey of Vision-driven Datasets and Studies](https://mlsanity.com/paper/2507.06513/) (2025)
- [STAGNet: A Spatio-Temporal Graph and LSTM Framework for Accident Anticipation](https://mlsanity.com/paper/2508.15216/) (2025)
- [VAGNet: Vision-based Accident Anticipation with Global Features](https://mlsanity.com/paper/2604.09305/) (2026)
- [Recent Advances in Graph-based Machine Learning for Applications in Smart Urban Transportation Systems](https://mlsanity.com/paper/2306.01282/) (2023)
- [STDG: Semi-Teacher-Student Training Paradigram for Depth-guided One-stage Scene Graph Generation](https://mlsanity.com/paper/2309.08179/) (2023)
- [Collision Probability Distribution Estimation via Temporal Difference Learning](https://mlsanity.com/paper/2407.20000/) (2024)
- [A transition towards virtual representations of visual scenes](https://mlsanity.com/paper/2410.07987/) (2024)
- [MsFIN: Multi-scale Feature Interaction Network for Traffic Accident Anticipation](https://mlsanity.com/paper/2509.19227/) (2025)
- [FLaRA: Predicting Future Latent Representations for Accident Anticipation](https://mlsanity.com/paper/2606.14380/) (2026)
- [hia-gat: A Heterogeneous Interaction-Aware Graph Attention Network For Frame-Level Traffic Conflict Risk Prediction On Freeways](https://mlsanity.com/paper/2606.27577/) (2026)

## Abstract (arXiv)

In autonomous vehicles (AVs), early warning systems rely on collision prediction to ensure occupant safety. However, state-of-the-art methods using deep convolutional networks either fail at modeling collisions or are too expensive/slow, making them less suitable for deployment on AV edge hardware. To address these limitations, we propose sg2vec, a spatio-temporal scene-graph embedding methodology that uses Graph Neural Network (GNN) and Long Short-Term Memory (LSTM) layers to predict future collisions via visual scene perception. We demonstrate that sg2vec predicts collisions 8.11% more accurately and 39.07% earlier than the state-of-the-art method on synthesized datasets, and 29.47% more accurately on a challenging real-world collision dataset. We also show that sg2vec is better than the state-of-the-art at transferring knowledge from synthetic datasets to real-world driving datasets. Finally, we demonstrate that sg2vec performs inference 9.3x faster with an 88.0% smaller model, 32.4% less power, and 92.8% less energy than the state-of-the-art method on the industry-standard Nvidia DRIVE PX 2 platform, making it more suitable for implementation on the edge.