# Uncertainty and Prediction Quality Estimation for Semantic Segmentation via Graph Neural Networks

arXiv: https://arxiv.org/abs/2409.11373 · HTML: https://mlsanity.com/paper/2409.11373/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 42 of its 61 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding](https://mlsanity.com/paper/1511.02680/) (2015)
- [Wider or Deeper: Revisiting the ResNet Model for Visual Recognition](https://mlsanity.com/paper/1611.10080/) (2016)
- [Improving Semantic Segmentation via Video Propagation and Label Relaxation](https://mlsanity.com/paper/1812.01593/) (2018)
- [Entropy Maximization and Meta Classification for Out-Of-Distribution Detection in Semantic Segmentation](https://mlsanity.com/paper/2012.06575/) (2020)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Hyperbolic Image Segmentation](https://mlsanity.com/paper/2203.05898/) (2022)
- [Prediction Error Meta Classification in Semantic Segmentation: Detection via Aggregated Dispersion Measures of Softmax Probabilities](https://mlsanity.com/paper/1811.00648/) (2018)
- [Leveraging Uncertainty Estimates for Predicting Segmentation Quality](https://mlsanity.com/paper/1807.00502/) (2018)
- [Dynamic Occupancy Grid Prediction for Urban Autonomous Driving: A Deep Learning Approach with Fully Automatic Labeling](https://mlsanity.com/paper/1705.08781/) (2017)
- [MetaBox+: A new Region Based Active Learning Method for Semantic Segmentation using Priority Maps](https://mlsanity.com/paper/2010.01884/) (2020)
- [Time-Dynamic Estimates of the Reliability of Deep Semantic Segmentation Networks](https://mlsanity.com/paper/1911.05075/) (2019)
- [Detection and Retrieval of Out-of-Distribution Objects in Semantic Segmentation](https://mlsanity.com/paper/2005.06831/) (2020)
- [Automated Detection of Label Errors in Semantic Segmentation Datasets via Deep Learning and Uncertainty Quantification](https://mlsanity.com/paper/2207.06104/) (2022)
- [Uncertainty Measures and Prediction Quality Rating for the Semantic Segmentation of Nested Multi Resolution Street Scene Images](https://mlsanity.com/paper/1904.04516/) (2019)
- [MetaDetect: Uncertainty Quantification and Prediction Quality Estimates for Object Detection](https://mlsanity.com/paper/2010.01695/) (2020)
- [Towards increased trustworthiness of deep learning segmentation methods on cardiac MRI](https://mlsanity.com/paper/1809.10430/) (2018)
- [End-to-End Deep Learning of Lane Detection and Path Prediction for Real-Time Autonomous Driving](https://mlsanity.com/paper/2102.04738/) (2021)

## Abstract (arXiv)

When employing deep neural networks (DNNs) for semantic segmentation in safety-critical applications like automotive perception or medical imaging, it is important to estimate their performance at runtime, e.g. via uncertainty estimates or prediction quality estimates. Previous works mostly performed uncertainty estimation on pixel-level. In a line of research, a connected-component-wise (segment-wise) perspective was taken, approaching uncertainty estimation on an object-level by performing so-called meta classification and regression to estimate uncertainty and prediction quality, respectively. In those works, each predicted segment is considered individually to estimate its uncertainty or prediction quality. However, the neighboring segments may provide additional hints on whether a given predicted segment is of high quality, which we study in the present work. On the basis of uncertainty indicating metrics on segment-level, we use graph neural networks (GNNs) to model the relationship of a given segment's quality as a function of the given segment's metrics as well as those of its neighboring segments. We compare different GNN architectures and achieve a notable performance improvement.