# Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilities

arXiv: https://arxiv.org/abs/1811.00648 · HTML: https://mlsanity.com/paper/1811.00648/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2018.

## Built on by (most cited first, 30 of 54)

- [SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation](https://mlsanity.com/paper/2104.14812/) (2021)
- [Pixel-wise Anomaly Detection in Complex Driving Scenes](https://mlsanity.com/paper/2103.05445/) (2021)
- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [Unmasking Anomalies in Road-Scene Segmentation](https://mlsanity.com/paper/2307.13316/) (2023)
- [On Advantages of Mask-level Recognition for Outlier-aware Segmentation](https://mlsanity.com/paper/2301.03407/) (2023)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [Automated Detection of Label Errors in Semantic Segmentation Datasets via Deep Learning and Uncertainty Quantification](https://mlsanity.com/paper/2207.06104/) (2022)
- [Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects](https://mlsanity.com/paper/2210.02074/) (2022)
- [Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation](https://mlsanity.com/paper/2309.04573/) (2023)
- [Gradient-Based Quantification of Epistemic Uncertainty for Deep Object Detectors](https://mlsanity.com/paper/2107.04517/) (2021)
- [Towards Unsupervised Open World Semantic Segmentation](https://mlsanity.com/paper/2201.01073/) (2022)
- [MetaToken: Detecting Hallucination in Image Descriptions by Meta Classification](https://mlsanity.com/paper/2405.19186/) (2024)
- [Out-of-Distribution Segmentation in Autonomous Driving: Problems and State of the Art](https://mlsanity.com/paper/2503.08695/) (2025)
- [Pixel-wise Gradient Uncertainty for Convolutional Neural Networks applied to Out-of-Distribution Segmentation](https://mlsanity.com/paper/2303.06920/) (2023)
- [Reducing Texture Bias of Deep Neural Networks via Edge Enhancing Diffusion](https://mlsanity.com/paper/2402.09530/) (2024)
- [WDiscOOD: Out-of-Distribution Detection via Whitened Linear Discriminant Analysis](https://mlsanity.com/paper/2303.07543/) (2023)
- [Have We Ever Encountered This Before? Retrieving Out-of-Distribution Road Obstacles from Driving Scenes](https://mlsanity.com/paper/2309.04302/) (2023)
- [See Yourself in Others: Attending Multiple Tasks for Own Failure Detection](https://mlsanity.com/paper/2110.02549/) (2021)
- [Enabling Verification of Deep Neural Networks in Perception Tasks Using Fuzzy Logic and Concept Embeddings](https://mlsanity.com/paper/2201.00572/) (2022)
- [Structural-Based Uncertainty in Deep Learning Across Anatomical Scales: Analysis in White Matter Lesion Segmentation](https://mlsanity.com/paper/2311.08931/) (2023)
- [False Negative Reduction in Video Instance Segmentation using Uncertainty Estimates](https://mlsanity.com/paper/2106.14474/) (2021)
- [Beyond Voxel Prediction Uncertainty: Identifying brain lesions you can trust](https://mlsanity.com/paper/2209.10877/) (2022)
- [Compensation Learning in Semantic Segmentation](https://mlsanity.com/paper/2304.13428/) (2023)
- [Self-Aware Trajectory Prediction for Safe Autonomous Driving](https://mlsanity.com/paper/2305.09147/) (2023)
- [Does Redundancy in AI Perception Systems Help to Test for Super-Human Automated Driving Performance?](https://mlsanity.com/paper/2112.04758/) (2021)
- [Detecting and Learning the Unknown in Semantic Segmentation](https://mlsanity.com/paper/2202.08700/) (2022)
- [VL4AD: Vision-Language Models Improve Pixel-wise Anomaly Detection](https://mlsanity.com/paper/2409.17330/) (2024)
- [Validation of Simulation-Based Testing: Bypassing Domain Shift with Label-to-Image Synthesis](https://mlsanity.com/paper/2106.05549/) (2021)
- [Post-hoc Models for Performance Estimation of Machine Learning Inference](https://mlsanity.com/paper/2110.02459/) (2021)
- [False Negative Reduction in Semantic Segmentation under Domain Shift using Depth Estimation](https://mlsanity.com/paper/2207.03513/) (2022)

## Abstract (arXiv)

We present a method that "meta" classifies whether seg-ments predicted by a semantic segmentation neural networkintersect with the ground truth. For this purpose, we employ measures of dispersion for predicted pixel-wise class probability distributions, like classification entropy, that yield heat maps of the input scene's size. We aggregate these dispersion measures segment-wise and derive metrics that are well-correlated with the segment-wise IoU of prediction and ground truth. This procedure yields an almost plug and play post-processing tool to rate the prediction quality of semantic segmentation networks on segment level. This is especially relevant for monitoring neural networks in online applications like automated driving or medical imaging where reliability is of utmost importance. In our tests, we use publicly available state-of-the-art networks trained on the Cityscapes dataset and the BraTS2017 dataset and analyze the predictive power of different metrics as well as different sets of metrics. To this end, we compute logistic LASSO regression fits for the task of classifying IoU=0 vs. IoU>0 per segment and obtain AUROC values of up to 91.55%. We complement these tests with linear regression fits to predict the segment-wise IoU and obtain prediction standard deviations of down to 0.130 as well as R2 values of up to 84.15%. We show that these results clearly outperform standard approaches.