# Perception Datasets for Anomaly Detection in Autonomous Driving: A Survey

arXiv: https://arxiv.org/abs/2302.02790 · HTML: https://mlsanity.com/paper/2302.02790/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 71 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [VOS: Learning What You Don't Know by Virtual Outlier Synthesis](https://mlsanity.com/paper/2202.01197/) (2022)
- [One Million Scenes for Autonomous Driving: ONCE Dataset](https://mlsanity.com/paper/2106.11037/) (2021)
- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)
- [Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior](https://mlsanity.com/paper/2112.05077/) (2021)
- [Detecting the Unexpected via Image Resynthesis](https://mlsanity.com/paper/1904.07595/) (2019)
- [The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation](https://mlsanity.com/paper/1904.03215/) (2019)
- [Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles](https://mlsanity.com/paper/1609.04653/) (2016)
- [CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2203.07724/) (2022)
- [KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients](https://mlsanity.com/paper/2204.13683/) (2022)
- [Anomaly Detection in Autonomous Driving: A Survey](https://mlsanity.com/paper/2204.07974/) (2022)
- [Identifying Unknown Instances for Autonomous Driving](https://mlsanity.com/paper/1910.11296/) (2019)
- [Rain rendering for evaluating and improving robustness to bad weather](https://mlsanity.com/paper/2009.03683/) (2020)
- [Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift](https://mlsanity.com/paper/1908.01098/) (2019)
- [SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving](https://mlsanity.com/paper/2106.11118/) (2021)
- [Description of Corner Cases in Automated Driving: Goals and Challenges](https://mlsanity.com/paper/2109.09607/) (2021)
- [An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Driving](https://mlsanity.com/paper/2103.03678/) (2021)
- [Dense open-set recognition with synthetic outliers generated by Real NVP](https://mlsanity.com/paper/2011.11094/) (2020)
- [MUAD: Multiple Uncertainties for Autonomous Driving, a benchmark for multiple uncertainty types and tasks](https://mlsanity.com/paper/2203.01437/) (2022)
- [Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects](https://mlsanity.com/paper/2210.02074/) (2022)
- [One Ontology to Rule Them All: Corner Case Scenarios for Autonomous Driving](https://mlsanity.com/paper/2209.00342/) (2022)
- [Towards Unsupervised Open World Semantic Segmentation](https://mlsanity.com/paper/2201.01073/) (2022)
- [Ad-datasets: a meta-collection of data sets for autonomous driving](https://mlsanity.com/paper/2202.01909/) (2022)
- [Space, Time, and Interaction: A Taxonomy of Corner Cases in Trajectory Datasets for Automated Driving](https://mlsanity.com/paper/2210.08885/) (2022)

## Built on by (most cited first, 16 of 16)

- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2405.07865/) (2024)
- [Out-of-Distribution Segmentation in Autonomous Driving: Problems and State of the Art](https://mlsanity.com/paper/2503.08695/) (2025)
- [Collaborative Perception Datasets in Autonomous Driving: A Survey](https://mlsanity.com/paper/2404.14022/) (2024)
- [Spotting the Unexpected (STU): A 3D LiDAR Dataset for Anomaly Segmentation in Autonomous Driving](https://mlsanity.com/paper/2505.02148/) (2025)
- [Exploring the Potential of World Models for Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2308.05701/) (2023)
- [Unveiling Objects with SOLA: An Annotation-Free Image Search on the Object Level for Automotive Data Sets](https://mlsanity.com/paper/2312.01860/) (2023)
- [A Survey on Video Anomaly Detection via Deep Learning: Human, Vehicle, and Environment](https://mlsanity.com/paper/2508.14203/) (2025)
- [UMAD: Unsupervised Mask-Level Anomaly Detection for Autonomous Driving](https://mlsanity.com/paper/2406.06370/) (2024)
- [Conditioning Latent-Space Clusters for Real-World Anomaly Classification](https://mlsanity.com/paper/2309.09676/) (2023)
- [Label-Free Model Failure Detection for Lidar-based Point Cloud Segmentation](https://mlsanity.com/paper/2407.14306/) (2024)
- [My Precious Crash Data: Barriers and Opportunities in Encouraging Autonomous Driving Companies to Share Safety-Critical Data](https://mlsanity.com/paper/2504.17792/) (2025)
- [Utilizing dataset affinity prediction in object detection to assess training data](https://mlsanity.com/paper/2311.09768/) (2023)
- [Dataset Safety in Autonomous Driving: Requirements, Risks, and Assurance](https://mlsanity.com/paper/2511.08439/) (2025)
- [AD4AD: Benchmarking Visual Anomaly Detection Models for Safer Autonomous Driving](https://mlsanity.com/paper/2604.15291/) (2026)
- [Real-World On-Vehicle Evaluation of Embedding-Based Anomaly Detection](https://mlsanity.com/paper/2605.19744/) (2026)

## Abstract (arXiv)

Deep neural networks (DNN) which are employed in perception systems for autonomous driving require a huge amount of data to train on, as they must reliably achieve high performance in all kinds of situations. However, these DNN are usually restricted to a closed set of semantic classes available in their training data, and are therefore unreliable when confronted with previously unseen instances. Thus, multiple perception datasets have been created for the evaluation of anomaly detection methods, which can be categorized into three groups: real anomalies in real-world, synthetic anomalies augmented into real-world and completely synthetic scenes. This survey provides a structured and, to the best of our knowledge, complete overview and comparison of perception datasets for anomaly detection in autonomous driving. Each chapter provides information about tasks and ground truth, context information, and licenses. Additionally, we discuss current weaknesses and gaps in existing datasets to underline the importance of developing further data.