# PeSOTIF: a Challenging Visual Dataset for Perception SOTIF Problems in Long-tail Traffic Scenarios arXiv: https://arxiv.org/abs/2211.03402 ยท HTML: https://mlsanity.com/paper/2211.03402/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 27 of its 44 references that are in the map. - First posted on arXiv 7 Nov 2022. ## Builds on (most cited first, 27 of 27) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015) - [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016) - [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017) - [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013) - [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018) - [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020) - [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019) - [Attentive Generative Adversarial Network for Raindrop Removal from a Single Image](https://mlsanity.com/paper/1711.10098/) (2017) - [A2D2: Audi Autonomous Driving Dataset](https://mlsanity.com/paper/2004.06320/) (2020) - [Getting to Know Low-light Images with The Exclusively Dark Dataset](https://mlsanity.com/paper/1805.11227/) (2018) - [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018) - [Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow](https://mlsanity.com/paper/1802.07095/) (2018) - [A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.10671/) (2020) - [RADIATE: A Radar Dataset for Automotive Perception in Bad Weather](https://mlsanity.com/paper/2010.09076/) (2020) - [The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes](https://mlsanity.com/paper/1903.01568/) (2019) - [Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty](https://mlsanity.com/paper/1711.09026/) (2017) - [Probabilistic Object Detection: Definition and Evaluation](https://mlsanity.com/paper/1811.10800/) (2018) - [Evaluating Merging Strategies for Sampling-based Uncertainty Techniques in Object Detection](https://mlsanity.com/paper/1809.06006/) (2018) - [Towards a Framework to Manage Perceptual Uncertainty for Safe Automated Driving](https://mlsanity.com/paper/1903.03438/) (2019) - [Training and Testing Object Detectors with Virtual Images](https://mlsanity.com/paper/1712.08470/) (2017) - [Road Damage Detection using Deep Ensemble Learning](https://mlsanity.com/paper/2011.00728/) (2020) - [Dirichlet uncertainty wrappers for actionable algorithm accuracy accountability and auditability](https://mlsanity.com/paper/1912.12628/) (2019) ## Built on by (most cited first, 8 of 8) - [Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities](https://mlsanity.com/paper/2509.08302/) (2025) - [SOTIF Entropy: Online SOTIF Risk Quantification and Mitigation for Autonomous Driving](https://mlsanity.com/paper/2211.04009/) (2022) - [DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models](https://mlsanity.com/paper/2505.07084/) (2025) - [Stereo-based 3D Anomaly Object Detection for Autonomous Driving: A New Dataset and Baseline](https://mlsanity.com/paper/2507.09214/) (2025) - [Simulation-Based Performance Evaluation of 3D Object Detection Methods with Deep Learning for a LiDAR Point Cloud Dataset in a SOTIF-related Use Case](https://mlsanity.com/paper/2503.03548/) (2025) - [A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions](https://mlsanity.com/paper/2601.22830/) (2026) - [A Systematic Literature Review on Safety of the Intended Functionality for Automated Driving Systems](https://mlsanity.com/paper/2503.02498/) (2025) - [What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs](https://mlsanity.com/paper/2606.01624/) (2026) ## Abstract (arXiv) Perception algorithms in autonomous driving systems confront great challenges in long-tail traffic scenarios, where the problems of Safety of the Intended Functionality (SOTIF) could be triggered by the algorithm performance insufficiencies and dynamic operational environment. However, such scenarios are not systematically included in current open-source datasets, and this paper fills the gap accordingly. Based on the analysis and enumeration of trigger conditions, a high-quality diverse dataset is released, including various long-tail traffic scenarios collected from multiple resources. Considering the development of probabilistic object detection (POD), this dataset marks trigger sources that may cause perception SOTIF problems in the scenarios as key objects. In addition, an evaluation protocol is suggested to verify the effectiveness of POD algorithms in identifying the key objects via uncertainty. The dataset never stops expanding, and the first batch of open-source data includes 1126 frames with an average of 2.27 key objects and 2.47 normal objects in each frame. To demonstrate how to use this dataset for SOTIF research, this paper further quantifies the perception SOTIF entropy to confirm whether a scenario is unknown and unsafe for a perception system. The experimental results show that the quantified entropy can effectively and efficiently reflect the failure of the perception algorithm.