# Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark

arXiv: https://arxiv.org/abs/2212.09381 · HTML: https://mlsanity.com/paper/2212.09381/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 19 Dec 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [University-1652: A Multi-view Multi-source Benchmark for Drone-based Geo-localization](https://mlsanity.com/paper/2002.12186/) (2020)
- [DRAMA: Joint Risk Localization and Captioning in Driving](https://mlsanity.com/paper/2209.10767/) (2022)
- [Uncertainty-based Traffic Accident Anticipation with Spatio-Temporal Relational Learning](https://mlsanity.com/paper/2008.00334/) (2020)
- [SUTD-TrafficQA: A Question Answering Benchmark and an Efficient Network for Video Reasoning over Traffic Events](https://mlsanity.com/paper/2103.15538/) (2021)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [Predicting the Driver's Focus of Attention: the DR(eye)VE Project](https://mlsanity.com/paper/1705.03854/) (2017)
- [Unsupervised Traffic Accident Detection in First-Person Videos](https://mlsanity.com/paper/1903.00618/) (2019)
- [DADA: Driver Attention Prediction in Driving Accident Scenarios](https://mlsanity.com/paper/1912.12148/) (2019)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [Invariant Grounding for Video Question Answering](https://mlsanity.com/paper/2206.02349/) (2022)
- [Predicting Driver Attention in Critical Situations](https://mlsanity.com/paper/1711.06406/) (2017)
- [DADA-2000: Can Driving Accident be Predicted by Driver Attention? Analyzed by A Benchmark](https://mlsanity.com/paper/1904.12634/) (2019)
- [A Deep Multi-Level Network for Saliency Prediction](https://mlsanity.com/paper/1609.01064/) (2016)
- [Anticipating Traffic Accidents with Adaptive Loss and Large-scale Incident DB](https://mlsanity.com/paper/1804.02675/) (2018)
- [DRIVE: Deep Reinforced Accident Anticipation with Visual Explanation](https://mlsanity.com/paper/2107.10189/) (2021)
- [VehicleNet: Learning Robust Visual Representation for Vehicle Re-identification](https://mlsanity.com/paper/2004.06305/) (2020)
- [A Dynamic Spatial-temporal Attention Network for Early Anticipation of Traffic Accidents](https://mlsanity.com/paper/2106.10197/) (2021)
- [PiPa: Pixel- and Patch-wise Self-supervised Learning for Domain Adaptative Semantic Segmentation](https://mlsanity.com/paper/2211.07609/) (2022)
- [CADP: A Novel Dataset for CCTV Traffic Camera based Accident Analysis](https://mlsanity.com/paper/1809.05782/) (2018)
- [Graph-Based Spatial-Temporal Convolutional Network for Vehicle Trajectory Prediction in Autonomous Driving](https://mlsanity.com/paper/2109.12764/) (2021)
- [Parameter-Efficient Person Re-identification in the 3D Space](https://mlsanity.com/paper/2006.04569/) (2020)
- [Behavioral Intention Prediction in Driving Scenes: A Survey](https://mlsanity.com/paper/2211.00385/) (2022)
- [MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1912.07773/) (2019)
- [Spatio-Temporal Scene-Graph Embedding for Autonomous Vehicle Collision Prediction](https://mlsanity.com/paper/2111.06123/) (2021)
- [Equivariant and Invariant Grounding for Video Question Answering](https://mlsanity.com/paper/2207.12783/) (2022)
- [Coupling Intent and Action for Pedestrian Crossing Behavior Prediction](https://mlsanity.com/paper/2105.04133/) (2021)

## Built on by (most cited first, 16 of 16)

- [Sora as a World Model? A Complete Survey on Text-to-Video Generation](https://mlsanity.com/paper/2403.05131/) (2024)
- [Vision-Based Traffic Accident Detection and Anticipation: A Survey](https://mlsanity.com/paper/2308.15985/) (2023)
- [Nexar Dashcam Collision Prediction Dataset and Challenge](https://mlsanity.com/paper/2503.03848/) (2025)
- [VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understanding](https://mlsanity.com/paper/2507.09815/) (2025)
- [Traffic-Domain Video Question Answering with Automatic Captioning](https://mlsanity.com/paper/2307.09636/) (2023)
- [Accident Anticipation via Temporal Occurrence Prediction](https://mlsanity.com/paper/2510.22260/) (2025)
- [Simplifying Traffic Anomaly Detection with Video Foundation Models](https://mlsanity.com/paper/2507.09338/) (2025)
- [PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning](https://mlsanity.com/paper/2606.02443/) (2026)
- [Seeing before Observable: Potential Risk Reasoning in Autonomous Driving via Vision Language Models](https://mlsanity.com/paper/2511.22928/) (2025)
- [Understanding Real-World Traffic Safety through RoadSafe365 Benchmark](https://mlsanity.com/paper/2602.07212/) (2026)
- [RiskProp: Collision-Anchored Self-Supervised Risk Propagation for Early Accident Anticipation](https://mlsanity.com/paper/2603.27165/) (2026)
- [Deep Learning Advances in Vision-Based Traffic Accident Anticipation: A Comprehensive Review of Methods, Datasets, and Future Directions](https://mlsanity.com/paper/2505.07611/) (2025)
- [CrashChat: A Multimodal Large Language Model for Multitask Traffic Crash Video Analysis](https://mlsanity.com/paper/2512.18878/) (2025)
- [VAGNet: Vision-based Accident Anticipation with Global Features](https://mlsanity.com/paper/2604.09305/) (2026)
- [VZCrash: A Large-Scale IMU Dataset of Ego-Vehicle Crashes](https://mlsanity.com/paper/2606.06074/) (2026)
- [Progressive Risk Estimation for Accident Anticipation](https://mlsanity.com/paper/2609.32811/) (2026)

## Abstract (arXiv)

Traffic accident prediction in driving videos aims to provide an early warning of the accident occurrence, and supports the decision making of safe driving systems. Previous works usually concentrate on the spatial-temporal correlation of object-level context, while they do not fit the inherent long-tailed data distribution well and are vulnerable to severe environmental change. In this work, we propose a Cognitive Accident Prediction (CAP) method that explicitly leverages human-inspired cognition of text description on the visual observation and the driver attention to facilitate model training. In particular, the text description provides a dense semantic description guidance for the primary context of the traffic scene, while the driver attention provides a traction to focus on the critical region closely correlating with safe driving. CAP is formulated by an attentive text-to-vision shift fusion module, an attentive scene context transfer module, and the driver attention guided accident prediction module. We leverage the attention mechanism in these modules to explore the core semantic cues for accident prediction. In order to train CAP, we extend an existing self-collected DADA-2000 dataset (with annotated driver attention for each frame) with further factual text descriptions for the visual observations before the accidents. Besides, we construct a new large-scale benchmark consisting of 11,727 in-the-wild accident videos with over 2.19 million frames (named as CAP-DATA) together with labeled fact-effect-reason-introspection description and temporal accident frame label. Based on extensive experiments, the superiority of CAP is validated compared with state-of-the-art approaches. The code, CAP-DATA, and all results will be released in urlhttps://github.com/JWFanggit/LOTVS-CAP.