# Uncertainty Measures and Prediction Quality Rating for the Semantic Segmentation of Nested Multi Resolution Street Scene Images

arXiv: https://arxiv.org/abs/1904.04516 · HTML: https://mlsanity.com/paper/1904.04516/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2019.

## Built on by (most cited first, 18 of 18)

- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations](https://mlsanity.com/paper/2104.05788/) (2021)
- [Automated Detection of Label Errors in Semantic Segmentation Datasets via Deep Learning and Uncertainty Quantification](https://mlsanity.com/paper/2207.06104/) (2022)
- [Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects](https://mlsanity.com/paper/2210.02074/) (2022)
- [Gradient-Based Quantification of Epistemic Uncertainty for Deep Object Detectors](https://mlsanity.com/paper/2107.04517/) (2021)
- [Towards Unsupervised Open World Semantic Segmentation](https://mlsanity.com/paper/2201.01073/) (2022)
- [MetaToken: Detecting Hallucination in Image Descriptions by Meta Classification](https://mlsanity.com/paper/2405.19186/) (2024)
- [Have We Ever Encountered This Before? Retrieving Out-of-Distribution Road Obstacles from Driving Scenes](https://mlsanity.com/paper/2309.04302/) (2023)
- [Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space](https://mlsanity.com/paper/2404.08923/) (2024)
- [False Positive Detection and Prediction Quality Estimation for LiDAR Point Cloud Segmentation](https://mlsanity.com/paper/2110.15681/) (2021)
- [Detecting Novelties with Empty Classes](https://mlsanity.com/paper/2305.00983/) (2023)
- [Uncertainty Quantification and Resource-Demanding Computer Vision Applications of Deep Learning](https://mlsanity.com/paper/2205.14917/) (2022)
- [Uncertainty Quantification in Deep Neural Networks through Statistical Inference on Latent Space](https://mlsanity.com/paper/2305.10840/) (2023)
- [Defensive Perception: Estimation and Monitoring of Neural Network Performance under Deployment](https://mlsanity.com/paper/2308.06299/) (2023)
- [Improving EO Foundation Models with Confidence Assessment for enhanced Semantic segmentation](https://mlsanity.com/paper/2406.18279/) (2024)
- [Uncertainty and Prediction Quality Estimation for Semantic Segmentation via Graph Neural Networks](https://mlsanity.com/paper/2409.11373/) (2024)
- [Neural Network Meta Classifier: Improving the Reliability of Anomaly Segmentation](https://mlsanity.com/paper/2412.10765/) (2024)

## Abstract (arXiv)

In the semantic segmentation of street scenes the reliability of the prediction and therefore uncertainty measures are of highest interest. We present a method that generates for each input image a hierarchy of nested crops around the image center and presents these, all re-scaled to the same size, to a neural network for semantic segmentation. The resulting softmax outputs are then post processed such that we can investigate mean and variance over all image crops as well as mean and variance of uncertainty heat maps obtained from pixel-wise uncertainty measures, like the entropy, applied to each crop's softmax output. In our tests, we use the publicly available DeepLabv3+ MobilenetV2 network (trained on the Cityscapes dataset) and demonstrate that the incorporation of crops improves the quality of the prediction and that we obtain more reliable uncertainty measures. These are then aggregated over predicted segments for either classifying between IoU=0 and IoU>0 (meta classification) or predicting the IoU via linear regression (meta regression). The latter yields reliable performance estimates for segmentation networks, in particular useful in the absence of ground truth. For the task of meta classification we obtain a classification accuracy of 81.93% and an AUROC of 89.89%. For meta regression we obtain an R2 value of 84.77%. These results yield significant improvements compared to other approaches.