# On Calibrating Semantic Segmentation Models: Analyses and An Algorithm

arXiv: https://arxiv.org/abs/2212.12053 · HTML: https://mlsanity.com/paper/2212.12053/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 59 references that are in the map.
- First posted on arXiv 22 Dec 2022.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation](https://mlsanity.com/paper/1911.13273/) (2019)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Rethinking BiSeNet For Real-time Semantic Segmentation](https://mlsanity.com/paper/2104.13188/) (2021)
- [MSeg: A Composite Dataset for Multi-domain Semantic Segmentation](https://mlsanity.com/paper/2112.13762/) (2021)

## Built on by (most cited first, 25 of 25)

- [ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Model](https://mlsanity.com/paper/2304.03086/) (2023)
- [Conformal Semantic Image Segmentation: Post-hoc Quantification of Predictive Uncertainty](https://mlsanity.com/paper/2405.05145/) (2024)
- [Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding](https://mlsanity.com/paper/2403.17010/) (2024)
- [VSR-Net: Vessel-like Structure Rehabilitation Network with Graph Clustering](https://mlsanity.com/paper/2312.13116/) (2023)
- [Post-Hoc Reversal: Are We Selecting Models Prematurely?](https://mlsanity.com/paper/2404.07815/) (2024)
- [Improving Interpretability of Deep Active Learning for Flood Inundation Mapping Through Class Ambiguity Indices Using Multi-spectral Satellite Imagery](https://mlsanity.com/paper/2404.19043/) (2024)
- [ReliOcc: Towards Reliable Semantic Occupancy Prediction via Uncertainty Learning](https://mlsanity.com/paper/2409.18026/) (2024)
- [Understanding Calibration of Deep Neural Networks for Medical Image Classification](https://mlsanity.com/paper/2309.13132/) (2023)
- [AUGCAL: Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images](https://mlsanity.com/paper/2312.06106/) (2023)
- [A Self-Supervised Algorithm for Denoising Photoplethysmography Signals for Heart Rate Estimation from Wearables](https://mlsanity.com/paper/2307.05339/) (2023)
- [Probability calibration for precipitation nowcasting](https://mlsanity.com/paper/2510.00594/) (2025)
- [Uncertainty evaluation of segmentation models for Earth observation](https://mlsanity.com/paper/2510.19586/) (2025)
- [DA-Cal: Towards Cross-Domain Calibration in Semantic Segmentation](https://mlsanity.com/paper/2602.20860/) (2026)
- [PseudoCal: A Source-Free Approach to Unsupervised Uncertainty Calibration in Domain Adaptation](https://mlsanity.com/paper/2307.07489/) (2023)
- [Deep Bayesian segmentation for colon polyps: Well-calibrated predictions in medical imaging](https://mlsanity.com/paper/2407.16608/) (2024)
- [CARE: Confidence-aware Ratio Estimation for Medical Biomarkers](https://mlsanity.com/paper/2505.19585/) (2025)
- [Adaptive Confidence-Wise Loss for Improved Lens Structure Segmentation in AS-OCT](https://mlsanity.com/paper/2508.08705/) (2025)
- [Right Regions, Wrong Labels: Semantic Label Flips in Segmentation under Correlation Shift](https://mlsanity.com/paper/2604.13326/) (2026)
- [Multi-Rater Calibrated Segmentation Models](https://mlsanity.com/paper/2605.02437/) (2026)
- [Rethinking Post-Hoc Calibration in Semantic Segmentation](https://mlsanity.com/paper/2607.01902/) (2026)
- [Still image and spatial-temporal tomato data enabling detection, segmentation, tracking, and video-instance segmentation using strong and weak labels](https://mlsanity.com/paper/2607.14934/) (2026)
- [Leak-Free Cross-Validated Stacking with Per-Architecture Calibration for Sand-Boil Segmentation in Earthen Levees](https://mlsanity.com/paper/2607.25367/) (2026)
- [ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentation](https://mlsanity.com/paper/2608.01530/) (2026)
- [Generative Semantic Segmentation via an Observable Semantic-Image Interface and Hierarchical Generator Evidence Alignment](https://mlsanity.com/paper/2608.11537/) (2026)
- [Too Sure to Be Safe: Model Calibration for Reliable Log Anomaly Detection](https://mlsanity.com/paper/2608.17965/) (2026)

## Abstract (arXiv)

We study the problem of semantic segmentation calibration. Lots of solutions have been proposed to approach model miscalibration of confidence in image classification. However, to date, confidence calibration research on semantic segmentation is still limited. We provide a systematic study on the calibration of semantic segmentation models and propose a simple yet effective approach. First, we find that model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. Among them, prediction correctness, especially misprediction, is more important to miscalibration due to over-confidence. Next, we propose a simple, unifying, and effective approach, namely selective scaling, by separating correct/incorrect prediction for scaling and more focusing on misprediction logit smoothing. Then, we study popular existing calibration methods and compare them with selective scaling on semantic segmentation calibration. We conduct extensive experiments with a variety of benchmarks on both in-domain and domain-shift calibration and show that selective scaling consistently outperforms other methods.