# When, Where, and What? A Novel Benchmark for Accident Anticipation and Localization with Large Language Models

arXiv: https://arxiv.org/abs/2407.16277 · HTML: https://mlsanity.com/paper/2407.16277/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [T-GCN: A Temporal Graph ConvolutionalNetwork for Traffic Prediction](https://mlsanity.com/paper/1811.05320/) (2018)
- [GPT-Driver: Learning to Drive with GPT](https://mlsanity.com/paper/2310.01415/) (2023)
- [DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving](https://mlsanity.com/paper/2312.09245/) (2023)
- [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](https://mlsanity.com/paper/2211.09552/) (2022)
- [Uncertainty-based Traffic Accident Anticipation with Spatio-Temporal Relational Learning](https://mlsanity.com/paper/2008.00334/) (2020)
- [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024)
- [Unsupervised Traffic Accident Detection in First-Person Videos](https://mlsanity.com/paper/1903.00618/) (2019)
- [Anticipating Traffic Accidents with Adaptive Loss and Large-scale Incident DB](https://mlsanity.com/paper/1804.02675/) (2018)
- [DRIVE: Deep Reinforced Accident Anticipation with Visual Explanation](https://mlsanity.com/paper/2107.10189/) (2021)
- [TrafficGPT: Viewing, Processing and Interacting with Traffic Foundation Models](https://mlsanity.com/paper/2309.06719/) (2023)
- [A Dynamic Spatial-temporal Attention Network for Early Anticipation of Traffic Accidents](https://mlsanity.com/paper/2106.10197/) (2021)
- [Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localization](https://mlsanity.com/paper/1705.06560/) (2017)
- [Physical Backdoor Attacks to Lane Detection Systems in Autonomous Driving](https://mlsanity.com/paper/2203.00858/) (2022)
- [A Cognitive-Based Trajectory Prediction Approach for Autonomous Driving](https://mlsanity.com/paper/2402.19251/) (2024)
- [An Ethical Trajectory Planning Algorithm for Autonomous Vehicles](https://mlsanity.com/paper/2212.08577/) (2022)

## Built on by (most cited first, 4 of 4)

- [Real-time Accident Anticipation for Autonomous Driving Through Monocular Depth-Enhanced 3D Modeling](https://mlsanity.com/paper/2409.01256/) (2024)
- [When Will It Fail?: Anomaly to Prompt for Forecasting Future Anomalies in Time Series](https://mlsanity.com/paper/2506.23596/) (2025)
- [Accident Anticipation via Temporal Occurrence Prediction](https://mlsanity.com/paper/2510.22260/) (2025)
- [Deep Learning Advances in Vision-Based Traffic Accident Anticipation: A Comprehensive Review of Methods, Datasets, and Future Directions](https://mlsanity.com/paper/2505.07611/) (2025)

## Abstract (arXiv)

As autonomous driving systems increasingly become part of daily transportation, the ability to accurately anticipate and mitigate potential traffic accidents is paramount. Traditional accident anticipation models primarily utilizing dashcam videos are adept at predicting when an accident may occur but fall short in localizing the incident and identifying involved entities. Addressing this gap, this study introduces a novel framework that integrates Large Language Models (LLMs) to enhance predictive capabilities across multiple dimensions--what, when, and where accidents might occur. We develop an innovative chain-based attention mechanism that dynamically adjusts to prioritize high-risk elements within complex driving scenes. This mechanism is complemented by a three-stage model that processes outputs from smaller models into detailed multimodal inputs for LLMs, thus enabling a more nuanced understanding of traffic dynamics. Empirical validation on the DAD, CCD, and A3D datasets demonstrates superior performance in Average Precision (AP) and Mean Time-To-Accident (mTTA), establishing new benchmarks for accident prediction technology. Our approach not only advances the technological framework for autonomous driving safety but also enhances human-AI interaction, making predictive insights generated by autonomous systems more intuitive and actionable.