# Quantifying Model Uncertainty for Semantic Segmentation using Operators in the RKHS

arXiv: https://arxiv.org/abs/2211.01999 · HTML: https://mlsanity.com/paper/2211.01999/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 30 of 34)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts](https://mlsanity.com/paper/2006.09239/) (2020)
- [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [A Review on Deep Learning Techniques Applied to Semantic Segmentation](https://mlsanity.com/paper/1704.06857/) (2017)
- [Lightweight Probabilistic Deep Networks](https://mlsanity.com/paper/1805.11327/) (2018)
- [Understanding Softmax Confidence and Uncertainty](https://mlsanity.com/paper/2106.04972/) (2021)
- [On the Practicality of Deterministic Epistemic Uncertainty](https://mlsanity.com/paper/2107.00649/) (2021)
- [Efficient Uncertainty Estimation for Semantic Segmentation in Videos](https://mlsanity.com/paper/1807.11037/) (2018)
- [Uncertainty in the Variational Information Bottleneck](https://mlsanity.com/paper/1807.00906/) (2018)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling](https://mlsanity.com/paper/1505.07293/) (2015)
- [Deep Deterministic Uncertainty for Semantic Segmentation](https://mlsanity.com/paper/2111.00079/) (2021)
- [ReSeg: A Recurrent Neural Network-based Model for Semantic Segmentation](https://mlsanity.com/paper/1511.07053/) (2015)

## Built on by (most cited first, 2 of 2)

- [Globally Scalable Glacier Mapping by Deep Learning Matches Expert Delineation Accuracy](https://mlsanity.com/paper/2401.15113/) (2024)
- [Instance-wise Uncertainty for Class Imbalance in Semantic Segmentation](https://mlsanity.com/paper/2407.12609/) (2024)

## Abstract (arXiv)

Deep learning models for semantic segmentation are prone to poor performance in real-world applications due to the highly challenging nature of the task. Model uncertainty quantification (UQ) is one way to address this issue of lack of model trustworthiness by enabling the practitioner to know how much to trust a segmentation output. Current UQ methods in this application domain are mainly restricted to Bayesian based methods which are computationally expensive and are only able to extract central moments of uncertainty thereby limiting the quality of their uncertainty estimates. We present a simple framework for high-resolution predictive uncertainty quantification of semantic segmentation models that leverages a multi-moment functional definition of uncertainty associated with the model's feature space in the reproducing kernel Hilbert space (RKHS). The multiple uncertainty functionals extracted from this framework are defined by the local density dynamics of the model's feature space and hence automatically align themselves at the tail-regions of the intrinsic probability density function of the feature space (where uncertainty is the highest) in such a way that the successively higher order moments quantify the more uncertain regions. This leads to a significantly more accurate view of model uncertainty than conventional Bayesian methods. Moreover, the extraction of such moments is done in a single-shot computation making it much faster than Bayesian and ensemble approaches (that involve a high number of forward stochastic passes of the model to quantify its uncertainty). We demonstrate these advantages through experimental evaluations of our framework implemented over four different state-of-the-art model architectures that are trained and evaluated on two benchmark road-scene segmentation datasets (Camvid and Cityscapes).