# Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations

arXiv: https://arxiv.org/abs/2104.05788 · HTML: https://mlsanity.com/paper/2104.05788/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation](https://mlsanity.com/paper/1808.01200/) (2018)
- [PHiSeg: Capturing Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/1906.04045/) (2019)
- [Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty](https://mlsanity.com/paper/2006.06015/) (2020)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy](https://mlsanity.com/paper/1809.04430/) (2018)
- [Non-Parametric Calibration for Classification](https://mlsanity.com/paper/1906.04933/) (2019)
- [Multi-Atlas Segmentation of Biomedical Images: A Survey](https://mlsanity.com/paper/1412.3421/) (2014)
- [On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation](https://mlsanity.com/paper/1806.02562/) (2018)
- [Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference](https://mlsanity.com/paper/1909.13550/) (2019)
- [Uncertainty Measures and Prediction Quality Rating for the Semantic Segmentation of Nested Multi Resolution Street Scene Images](https://mlsanity.com/paper/1904.04516/) (2019)
- [TuNet: End-to-end Hierarchical Brain Tumor Segmentation using Cascaded Networks](https://mlsanity.com/paper/1910.05338/) (2019)

## Built on by (most cited first, 30 of 31)

- [A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods](https://mlsanity.com/paper/2310.06873/) (2023)
- [Tyche: Stochastic In-Context Learning for Medical Image Segmentation](https://mlsanity.com/paper/2401.13650/) (2024)
- [Label Cleaning Multiple Instance Learning: Refining Coarse Annotations on Single Whole-Slide Images](https://mlsanity.com/paper/2109.10778/) (2021)
- [Maximum Entropy on Erroneous Predictions (MEEP): Improving model calibration for medical image segmentation](https://mlsanity.com/paper/2112.12218/) (2021)
- [Trust your neighbours: Penalty-based constraints for model calibration](https://mlsanity.com/paper/2303.06268/) (2023)
- [On the Optimal Combination of Cross-Entropy and Soft Dice Losses for Lesion Segmentation with Out-of-Distribution Robustness](https://mlsanity.com/paper/2209.06078/) (2022)
- [Multi-Head Multi-Loss Model Calibration](https://mlsanity.com/paper/2303.01099/) (2023)
- [Learning to segment fetal brain tissue from noisy annotations](https://mlsanity.com/paper/2203.14962/) (2022)
- [Neighbor-Aware Calibration of Segmentation Networks with Penalty-Based Constraints](https://mlsanity.com/paper/2401.14487/) (2024)
- [Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results](https://mlsanity.com/paper/2505.08685/) (2025)
- [Paced-Curriculum Distillation with Prediction and Label Uncertainty for Image Segmentation](https://mlsanity.com/paper/2302.01049/) (2023)
- [Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels](https://mlsanity.com/paper/2303.16296/) (2023)
- [Deep Quantile Regression for Uncertainty Estimation in Unsupervised and Supervised Lesion Detection](https://mlsanity.com/paper/2109.09374/) (2021)
- [Boundary-weighted logit consistency improves calibration of segmentation networks](https://mlsanity.com/paper/2307.08163/) (2023)
- [Weakly Supervised Medical Image Segmentation With Soft Labels and Noise Robust Loss](https://mlsanity.com/paper/2209.08172/) (2022)
- [Class and Region-Adaptive Constraints for Network Calibration](https://mlsanity.com/paper/2403.12364/) (2024)
- [We Care Each Pixel: Calibrating on Medical Segmentation Model](https://mlsanity.com/paper/2503.05107/) (2025)
- [Context Label Learning: Improving Background Class Representations in Semantic Segmentation](https://mlsanity.com/paper/2212.08423/) (2022)
- [Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding](https://mlsanity.com/paper/2212.11511/) (2022)
- [Advancing Medical Image Segmentation: Morphology-Driven Learning with Diffusion Transformer](https://mlsanity.com/paper/2408.00347/) (2024)
- [Improving Uncertainty-Error Correspondence in Deep Bayesian Medical Image Segmentation](https://mlsanity.com/paper/2409.03470/) (2024)
- [A Lightweight and Extensible Cell Segmentation and Classification Model for Whole Slide Images](https://mlsanity.com/paper/2502.19217/) (2025)
- [SNCE: Geometry-Aware Supervision for Scalable Discrete Image Generation](https://mlsanity.com/paper/2603.15150/) (2026)
- [Beyond Boundary Noise: Aggregated Aleatoric Uncertainty Fails to Capture Presence Ambiguity in 3D Lung Nodule Segmentation](https://mlsanity.com/paper/2608.14766/) (2026)
- [Learning with Weak Annotations for Robust Maritime Obstacle Detection](https://mlsanity.com/paper/2206.13263/) (2022)
- [Can ultrasound confidence maps predict sonographers' labeling variability?](https://mlsanity.com/paper/2308.09433/) (2023)
- [OBSeg: Accurate and Fast Instance Segmentation Framework Using Segmentation Foundation Models with Oriented Bounding Box Prompts](https://mlsanity.com/paper/2401.08174/) (2024)
- [Good Enough? An Investigation on the Impact of Label Quality in Large-Scale Medical Datasets](https://mlsanity.com/paper/2505.20928/) (2025)
- [Assessing Pancreatic Ductal Adenocarcinoma Vascular Invasion: the PDACVI Benchmark](https://mlsanity.com/paper/2604.27582/) (2026)
- [Multi-Rater Calibrated Segmentation Models](https://mlsanity.com/paper/2605.02437/) (2026)

## Abstract (arXiv)

The task of image segmentation is inherently noisy due to ambiguities regarding the exact location of boundaries between anatomical structures. We argue that this information can be extracted from the expert annotations at no extra cost, and when integrated into state-of-the-art neural networks, it can lead to improved calibration between soft probabilistic predictions and the underlying uncertainty. We built upon label smoothing (LS) where a network is trained on 'blurred' versions of the ground truth labels which has been shown to be effective for calibrating output predictions. However, LS is not taking the local structure into account and results in overly smoothed predictions with low confidence even for non-ambiguous regions. Here, we propose Spatially Varying Label Smoothing (SVLS), a soft labeling technique that captures the structural uncertainty in semantic segmentation. SVLS also naturally lends itself to incorporate inter-rater uncertainty when multiple labelmaps are available. The proposed approach is extensively validated on four clinical segmentation tasks with different imaging modalities, number of classes and single and multi-rater expert annotations. The results demonstrate that SVLS, despite its simplicity, obtains superior boundary prediction with improved uncertainty and model calibration.