# Closer Look at the Uncertainty Estimation in Semantic Segmentation under Distributional Shift

arXiv: https://arxiv.org/abs/2106.00076 · HTML: https://mlsanity.com/paper/2106.00076/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 29 of 29)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation](https://mlsanity.com/paper/1911.13273/) (2019)
- [Learning Texture Invariant Representation for Domain Adaptation of Semantic Segmentation](https://mlsanity.com/paper/2003.00867/) (2020)
- [Benchmarking the Robustness of Semantic Segmentation Models](https://mlsanity.com/paper/1908.05005/) (2019)
- [A Study and Comparison of Human and Deep Learning Recognition Performance Under Visual Distortions](https://mlsanity.com/paper/1705.02498/) (2017)
- [Rethinking Domain Generalization Baselines](https://mlsanity.com/paper/2101.09060/) (2021)
- [Keep it Simple: Image Statistics Matching for Domain Adaptation](https://mlsanity.com/paper/2005.12551/) (2020)
- [Leveraging Uncertainties for Deep Multi-modal Object Detection in Autonomous Driving](https://mlsanity.com/paper/2002.00216/) (2020)
- [Robustness in Compressed Neural Networks for Object Detection](https://mlsanity.com/paper/2102.05509/) (2021)
- [EDF: Ensemble, Distill, and Fuse for Easy Video Labeling](https://mlsanity.com/paper/1812.03626/) (2018)

## Built on by (most cited first, 8 of 8)

- [Conformal Semantic Image Segmentation: Post-hoc Quantification of Predictive Uncertainty](https://mlsanity.com/paper/2405.05145/) (2024)
- [Extracting Uncertainty Estimates from Mixtures of Experts for Semantic Segmentation](https://mlsanity.com/paper/2509.04816/) (2025)
- [On the Calibration of Uncertainty Estimation in LiDAR-based Semantic Segmentation](https://mlsanity.com/paper/2308.02248/) (2023)
- [Learning to Generate Training Datasets for Robust Semantic Segmentation](https://mlsanity.com/paper/2308.02535/) (2023)
- [Robust Object Detection with Multi-input Multi-output Faster R-CNN](https://mlsanity.com/paper/2111.13065/) (2021)
- [On the calibration of underrepresented classes in LiDAR-based semantic segmentation](https://mlsanity.com/paper/2210.06811/) (2022)
- [Uncertainty evaluation of segmentation models for Earth observation](https://mlsanity.com/paper/2510.19586/) (2025)
- [Instance-wise Uncertainty for Class Imbalance in Semantic Segmentation](https://mlsanity.com/paper/2407.12609/) (2024)

## Abstract (arXiv)

While recent computer vision algorithms achieve impressive performance on many benchmarks, they lack robustness - presented with an image from a different distribution, (e.g. weather or lighting conditions not considered during training), they may produce an erroneous prediction. Therefore, it is desired that such a model will be able to reliably predict its confidence measure. In this work, uncertainty estimation for the task of semantic segmentation is evaluated under a varying level of domain shift: in a cross-dataset setting and when adapting a model trained on data from the simulation. It was shown that simple color transformations already provide a strong baseline, comparable to using more sophisticated style-transfer data augmentation. Further, by constructing an ensemble consisting of models using different backbones and/or augmentation methods, it was possible to improve significantly model performance in terms of overall accuracy and uncertainty estimation under the domain shift setting. The Expected Calibration Error (ECE) on challenging GTA to Cityscapes adaptation was reduced from 4.05 to the competitive value of 1.1. Further, an ensemble of models was utilized in the self-training setting to improve the pseudo-labels generation, which resulted in a significant gain in the final model accuracy, compared to the standard fine-tuning (without ensemble).