# A Dataset for Semantic Segmentation in the Presence of Unknowns

arXiv: https://arxiv.org/abs/2503.22309 · HTML: https://mlsanity.com/paper/2503.22309/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Builds on (most cited first, 26 of 26)

- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [The ApolloScape Open Dataset for Autonomous Driving and its Application](https://mlsanity.com/paper/1803.06184/) (2018)
- [IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments](https://mlsanity.com/paper/1811.10200/) (2018)
- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)
- [Detecting the Unexpected via Image Resynthesis](https://mlsanity.com/paper/1904.07595/) (2019)
- [The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation](https://mlsanity.com/paper/1904.03215/) (2019)
- [Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles](https://mlsanity.com/paper/1609.04653/) (2016)
- [RbA: Segmenting Unknown Regions Rejected by All](https://mlsanity.com/paper/2211.14293/) (2022)
- [Unmasking Anomalies in Road-Scene Segmentation](https://mlsanity.com/paper/2307.13316/) (2023)
- [On Advantages of Mask-level Recognition for Outlier-aware Segmentation](https://mlsanity.com/paper/2301.03407/) (2023)
- [Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding](https://mlsanity.com/paper/1901.01415/) (2019)
- [Outlier detection by ensembling uncertainty with negative objectness](https://mlsanity.com/paper/2402.15374/) (2024)
- [PixOOD: Pixel-Level Out-of-Distribution Detection](https://mlsanity.com/paper/2405.19882/) (2024)
- [Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects](https://mlsanity.com/paper/2210.02074/) (2022)
- [AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving](https://mlsanity.com/paper/2405.07865/) (2024)
- [IDD-X: A Multi-View Dataset for Ego-relative Important Object Localization and Explanation in Dense and Unstructured Traffic](https://mlsanity.com/paper/2404.08561/) (2024)
- [IDD-AW: A Benchmark for Safe and Robust Segmentation of Drive Scenes in Unstructured Traffic and Adverse Weather](https://mlsanity.com/paper/2311.14459/) (2023)
- [Hybrid Open-set Segmentation with Synthetic Negative Data](https://mlsanity.com/paper/2301.08555/) (2023)

## Built on by (most cited first, 2 of 2)

- [ProOOD: Prototype-Guided Out-of-Distribution 3D Occupancy Prediction](https://mlsanity.com/paper/2604.01081/) (2026)
- [SAFe: Segment-guided Aggregation of Feature Densities for Anomaly-aware Segmentation](https://mlsanity.com/paper/2609.24204/) (2026)

## Abstract (arXiv)

Before deployment in the real-world deep neural networks require thorough evaluation of how they handle both knowns, inputs represented in the training data, and unknowns (anomalies). This is especially important for scene understanding tasks with safety critical applications, such as in autonomous driving. Existing datasets allow evaluation of only knowns or unknowns - but not both, which is required to establish "in the wild" suitability of deep neural network models. To bridge this gap, we propose a novel anomaly segmentation dataset, ISSU, that features a diverse set of anomaly inputs from cluttered real-world environments. The dataset is twice larger than existing anomaly segmentation datasets, and provides a training, validation and test set for controlled in-domain evaluation. The test set consists of a static and temporal part, with the latter comprised of videos. The dataset provides annotations for both closed-set (knowns) and anomalies, enabling closed-set and open-set evaluation. The dataset covers diverse conditions, such as domain and cross-sensor shift, illumination variation and allows ablation of anomaly detection methods with respect to these variations. Evaluation results of current state-of-the-art methods confirm the need for improvements especially in domain-generalization, small and large object segmentation.