# Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding

arXiv: https://arxiv.org/abs/2407.05910 · HTML: https://mlsanity.com/paper/2407.05910/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [A Survey on Negative Transfer](https://mlsanity.com/paper/2009.00909/) (2020)
- [Relation-Aware Graph Attention Network for Visual Question Answering](https://mlsanity.com/paper/1903.12314/) (2019)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [Efficient Large-Scale Multi-Modal Classification](https://mlsanity.com/paper/1802.02892/) (2018)
- [Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations](https://mlsanity.com/paper/2305.06152/) (2023)
- [Scene-Graph Augmented Data-Driven Risk Assessment of Autonomous Vehicle Decisions](https://mlsanity.com/paper/2009.06435/) (2020)
- [Spatio-Temporal Scene-Graph Embedding for Autonomous Vehicle Collision Prediction](https://mlsanity.com/paper/2111.06123/) (2021)
- [Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysis](https://mlsanity.com/paper/2206.06367/) (2022)
- [An Empirical Study on Leveraging Scene Graphs for Visual Question Answering](https://mlsanity.com/paper/1907.12133/) (2019)
- [Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions](https://mlsanity.com/paper/2107.11508/) (2021)
- [Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph prediction](https://mlsanity.com/paper/2310.16494/) (2023)
- [roadscene2vec: A Tool for Extracting and Embedding Road Scene-Graphs](https://mlsanity.com/paper/2109.01183/) (2021)
- [Towards Traffic Scene Description: The Semantic Scene Graph](https://mlsanity.com/paper/2111.10196/) (2021)
- [Traffic-Domain Video Question Answering with Automatic Captioning](https://mlsanity.com/paper/2307.09636/) (2023)
- [Distribution-aware Goal Prediction and Conformant Model-based Planning for Safe Autonomous Driving](https://mlsanity.com/paper/2212.08729/) (2022)

## Built on by (most cited first, 10 of 10)

- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Review](https://mlsanity.com/paper/2506.06301/) (2025)
- [World Model-Based End-to-End Scene Generation for Accident Anticipation in Autonomous Driving](https://mlsanity.com/paper/2507.12762/) (2025)
- [Vision Technologies with Applications in Traffic Surveillance Systems: A Holistic Survey](https://mlsanity.com/paper/2412.00348/) (2024)
- [VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models](https://mlsanity.com/paper/2511.07299/) (2025)
- [Image Segmentation with Large Language Models: A Survey with Perspectives for Intelligent Transportation Systems](https://mlsanity.com/paper/2506.14096/) (2025)
- [Investigating Traffic Accident Detection Using Multimodal Large Language Models](https://mlsanity.com/paper/2509.19096/) (2025)
- [Structured Prompting and Multi-Agent Knowledge Distillation for Traffic Video Interpretation and Risk Inference](https://mlsanity.com/paper/2508.13439/) (2025)
- [Learning from the Unseen: Generative Data Augmentation for Geometric-Semantic Accident Anticipation](https://mlsanity.com/paper/2605.00051/) (2026)
- [Combining Hierarchical Cognitive Process with Process Supervision for Interpretable Scene Safety Understanding](https://mlsanity.com/paper/2609.26399/) (2026)

## Abstract (arXiv)

Recognizing a traffic accident is an essential part of any autonomous driving or road monitoring system. An accident can appear in a wide variety of forms, and understanding what type of accident is taking place may be useful to prevent it from recurring. This work focuses on classifying traffic scenes into specific accident types. We approach the problem by representing a traffic scene as a graph, where objects such as cars can be represented as nodes, and relative distances and directions between them as edges. This representation of a traffic scene is referred to as a scene graph, and can be used as input for an accident classifier. Better results are obtained with a classifier that fuses the scene graph input with visual and textual representations. This work introduces a multi-stage, multimodal pipeline that pre-processes videos of traffic accidents, encodes them as scene graphs, and aligns this representation with vision and language modalities before executing the classification task. When trained on 4 classes, our method achieves a balanced accuracy score of 57.77% on an (unbalanced) subset of the popular Detection of Traffic Anomaly (DoTA) benchmark, representing an increase of close to 5 percentage points from the case where scene graph information is not taken into account.