# The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation

arXiv: https://arxiv.org/abs/1904.03215 · HTML: https://mlsanity.com/paper/1904.03215/

## Key facts

- Cited by 122 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2019.

## Built on by (most cited first, 30 of 122)

- [VOS: Learning What You Don't Know by Virtual Outlier Synthesis](https://mlsanity.com/paper/2202.01197/) (2022)
- [OpenGAN: Open-Set Recognition via Open Data Generation](https://mlsanity.com/paper/2104.02939/) (2021)
- [Pixel-wise Anomaly Detection in Complex Driving Scenes](https://mlsanity.com/paper/2103.05445/) (2021)
- [CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2203.07724/) (2022)
- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes](https://mlsanity.com/paper/2111.12264/) (2021)
- [Standardized Max Logits: A Simple yet Effective Approach for Identifying Unexpected Road Obstacles in Urban-Scene Segmentation](https://mlsanity.com/paper/2107.11264/) (2021)
- [RbA: Segmenting Unknown Regions Rejected by All](https://mlsanity.com/paper/2211.14293/) (2022)
- [A Survey on Autonomous Driving Datasets: Statistics, Annotation Quality, and a Future Outlook](https://mlsanity.com/paper/2401.01454/) (2024)
- [On the Practicality of Deterministic Epistemic Uncertainty](https://mlsanity.com/paper/2107.00649/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Residual Pattern Learning for Pixel-wise Out-of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2211.14512/) (2022)
- [Unmasking Anomalies in Road-Scene Segmentation](https://mlsanity.com/paper/2307.13316/) (2023)
- [DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognition](https://mlsanity.com/paper/2207.02606/) (2022)
- [On Advantages of Mask-level Recognition for Outlier-aware Segmentation](https://mlsanity.com/paper/2301.03407/) (2023)
- [Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation](https://mlsanity.com/paper/2108.01634/) (2021)
- [Maskomaly:Zero-Shot Mask Anomaly Segmentation](https://mlsanity.com/paper/2305.16972/) (2023)
- [Outlier detection by ensembling uncertainty with negative objectness](https://mlsanity.com/paper/2402.15374/) (2024)
- [An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Driving](https://mlsanity.com/paper/2103.03678/) (2021)
- [MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities](https://mlsanity.com/paper/2405.17419/) (2024)
- [Reliability in Semantic Segmentation: Can We Use Synthetic Data?](https://mlsanity.com/paper/2312.09231/) (2023)
- [Open-World Semantic Segmentation Including Class Similarity](https://mlsanity.com/paper/2403.07532/) (2024)
- [PixOOD: Pixel-Level Out-of-Distribution Detection](https://mlsanity.com/paper/2405.19882/) (2024)
- [Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects](https://mlsanity.com/paper/2210.02074/) (2022)
- [Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation](https://mlsanity.com/paper/2309.04573/) (2023)
- [Perception Datasets for Anomaly Detection in Autonomous Driving: A Survey](https://mlsanity.com/paper/2302.02790/) (2023)
- [Balanced Energy Regularization Loss for Out-of-distribution Detection](https://mlsanity.com/paper/2306.10485/) (2023)
- [Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark](https://mlsanity.com/paper/2404.10760/) (2024)
- [Far Away in the Deep Space: Dense Nearest-Neighbor-Based Out-of-Distribution Detection](https://mlsanity.com/paper/2211.06660/) (2022)
- [UGainS: Uncertainty Guided Anomaly Instance Segmentation](https://mlsanity.com/paper/2308.02046/) (2023)

## Abstract (arXiv)

Deep learning has enabled impressive progress in the accuracy of semantic segmentation. Yet, the ability to estimate uncertainty and detect failure is key for safety-critical applications like autonomous driving. Existing uncertainty estimates have mostly been evaluated on simple tasks, and it is unclear whether these methods generalize to more complex scenarios. We present Fishyscapes, the first public benchmark for uncertainty estimation in a real-world task of semantic segmentation for urban driving. It evaluates pixel-wise uncertainty estimates towards the detection of anomalous objects in front of the vehicle. We~adapt state-of-the-art methods to recent semantic segmentation models and compare approaches based on softmax confidence, Bayesian learning, and embedding density. Our results show that anomaly detection is far from solved even for ordinary situations, while our benchmark allows measuring advancements beyond the state-of-the-art.