# When, Where, and What? A New Dataset for Anomaly Detection in Driving Videos

arXiv: https://arxiv.org/abs/2004.03044 · HTML: https://mlsanity.com/paper/2004.03044/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2020.

## Built on by (most cited first, 27 of 27)

- [DRAMA: Joint Risk Localization and Captioning in Driving](https://mlsanity.com/paper/2209.10767/) (2022)
- [Explanations in Autonomous Driving: A Survey](https://mlsanity.com/paper/2103.05154/) (2021)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control](https://mlsanity.com/paper/2412.11198/) (2024)
- [Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning](https://mlsanity.com/paper/2309.06597/) (2023)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [Spatio-Temporal Scene-Graph Embedding for Autonomous Vehicle Collision Prediction](https://mlsanity.com/paper/2111.06123/) (2021)
- [Nexar Dashcam Collision Prediction Dataset and Challenge](https://mlsanity.com/paper/2503.03848/) (2025)
- [RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding](https://mlsanity.com/paper/2304.08600/) (2023)
- [TimeRefine: Temporal Grounding with Time Refining Video LLM](https://mlsanity.com/paper/2412.09601/) (2024)
- [Coupling Intent and Action for Pedestrian Crossing Behavior Prediction](https://mlsanity.com/paper/2105.04133/) (2021)
- [IDD-X: A Multi-View Dataset for Ego-relative Important Object Localization and Explanation in Dense and Unstructured Traffic](https://mlsanity.com/paper/2404.08561/) (2024)
- [FT-AED: Benchmark Dataset for Early Freeway Traffic Anomalous Event Detection](https://mlsanity.com/paper/2406.15283/) (2024)
- [Ano-Graph: Learning Normal Scene Contextual Graphs to Detect Video Anomalies](https://mlsanity.com/paper/2103.10502/) (2021)
- [The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM](https://mlsanity.com/paper/2507.21649/) (2025)
- [Anomaly Crossing: New Horizons for Video Anomaly Detection as Cross-domain Few-shot Learning](https://mlsanity.com/paper/2112.06320/) (2021)
- [CycleCrash: A Dataset of Bicycle Collision Videos for Collision Prediction and Analysis](https://mlsanity.com/paper/2409.19942/) (2024)
- [A Survey on Datasets for Decision-making of Autonomous Vehicle](https://mlsanity.com/paper/2306.16784/) (2023)
- [POTLoc: Pseudo-Label Oriented Transformer for Point-Supervised Temporal Action Localization](https://mlsanity.com/paper/2310.13585/) (2023)
- [An Expert Ensemble for Detecting Anomalous Scenes, Interactions, and Behaviors in Autonomous Driving](https://mlsanity.com/paper/2502.16389/) (2025)
- [What Demands Attention in Urban Street Scenes? From Scene Understanding towards Road Safety: A Survey of Vision-driven Datasets and Studies](https://mlsanity.com/paper/2507.06513/) (2025)
- [STRIDE: Street View-based Environmental Feature Detection and Pedestrian Collision Prediction](https://mlsanity.com/paper/2308.13183/) (2023)
- [ADM-Loc: Actionness Distribution Modeling for Point-supervised Temporal Action Localization](https://mlsanity.com/paper/2311.15916/) (2023)
- [DAVE: Diverse Atomic Visual Elements Dataset with High Representation of Vulnerable Road Users in Complex and Unpredictable Environments](https://mlsanity.com/paper/2412.20042/) (2024)
- [Sparse Reasoning is Enough: Biological-Inspired Framework for Video Anomaly Detection with Large Pre-trained Models](https://mlsanity.com/paper/2511.17094/) (2025)
- [Metadata-Aware Multi-Prompt Reasoning for Zero-Shot Accident Understanding](https://mlsanity.com/paper/2606.12047/) (2026)
- [Asleep at the Wheel: JEPA's Limitations in Evaluating Novel Driving Data](https://mlsanity.com/paper/2608.01336/) (2026)

## Abstract (arXiv)

Video anomaly detection (VAD) has been extensively studied. However, research on egocentric traffic videos with dynamic scenes lacks large-scale benchmark datasets as well as effective evaluation metrics. This paper proposes traffic anomaly detection with a textitwhen-where-what pipeline to detect, localize, and recognize anomalous events from egocentric videos. We introduce a new dataset called Detection of Traffic Anomaly (DoTA) containing 4,677 videos with temporal, spatial, and categorical annotations. A new spatial-temporal area under curve (STAUC) evaluation metric is proposed and used with DoTA. State-of-the-art methods are benchmarked for two VAD-related tasks.Experimental results show STAUC is an effective VAD metric. To our knowledge, DoTA is the largest traffic anomaly dataset to-date and is the first supporting traffic anomaly studies across when-where-what perspectives. Our code and dataset can be found in: https://github.com/MoonBlvd/Detection-of-Traffic-Anomaly