# Beyond scalar losses: calibrating segmentation models via gradient vector field surgery

arXiv: https://arxiv.org/abs/2607.14338 · HTML: https://mlsanity.com/paper/2607.14338/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 15 Jul 2026.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](https://mlsanity.com/paper/2404.09556/) (2024)
- [Tversky loss function for image segmentation using 3D fully convolutional deep networks](https://mlsanity.com/paper/1706.05721/) (2017)
- [A Novel Focal Tversky loss function with improved Attention U-Net for lesion segmentation](https://mlsanity.com/paper/1810.07842/) (2018)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)
- [Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation](https://mlsanity.com/paper/1911.13273/) (2019)
- [The Numerics of GANs](https://mlsanity.com/paper/1705.10461/) (2017)
- [The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration](https://mlsanity.com/paper/2111.15430/) (2021)
- [Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation](https://mlsanity.com/paper/2102.04525/) (2021)
- [Combo Loss: Handling Input and Output Imbalance in Multi-Organ Segmentation](https://mlsanity.com/paper/1805.02798/) (2018)
- [Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations](https://mlsanity.com/paper/2104.05788/) (2021)
- [Primus: Enforcing Attention Usage for 3D Medical Image Segmentation](https://mlsanity.com/paper/2503.01835/) (2025)
- [Post Training Uncertainty Calibration of Deep Networks For Medical Image Segmentation](https://mlsanity.com/paper/2010.14290/) (2020)
- [Analysis of the MICCAI Brain Tumor Segmentation -- Metastases (BraTS-METS) 2025 Lighthouse Challenge: Brain Metastasis Segmentation on Pre- and Post-treatment MRI](https://mlsanity.com/paper/2504.12527/) (2025)
- [Calibrating the Dice loss to handle neural network overconfidence for biomedical image segmentation](https://mlsanity.com/paper/2111.00528/) (2021)
- [Towards increased trustworthiness of deep learning segmentation methods on cardiac MRI](https://mlsanity.com/paper/1809.10430/) (2018)
- [Trust your neighbours: Penalty-based constraints for model calibration](https://mlsanity.com/paper/2303.06268/) (2023)
- [Optimization with soft Dice can lead to a volumetric bias](https://mlsanity.com/paper/1911.02278/) (2019)
- [Neighbor-Aware Calibration of Segmentation Networks with Penalty-Based Constraints](https://mlsanity.com/paper/2401.14487/) (2024)
- [Average Calibration Error: A Differentiable Loss for Improved Reliability in Image Segmentation](https://mlsanity.com/paper/2403.06759/) (2024)
- [Pitfalls of topology-aware image segmentation](https://mlsanity.com/paper/2412.14619/) (2024)
- [Boundary-weighted logit consistency improves calibration of segmentation networks](https://mlsanity.com/paper/2307.08163/) (2023)
- [Calibrating Segmentation Networks with Margin-based Label Smoothing](https://mlsanity.com/paper/2209.09641/) (2022)

## Abstract (arXiv)

Region-based loss functions, such as the Dice loss, have established themselves as the de facto standard for highly class- and region-imbalanced segmentation tasks. However, models trained using region-based loss functions are notoriously miscalibrated and typically yield over-confident predictions. In medical imaging applications, such as defining tumor resection margins, this miscalibration is hindering clinical adoption. In this work, we outline a novel gradient perspective on this overconfidence and show how it affects region-based loss functions. We propose a "surgery" on the gradient vector field as a simple, yet effective intervention to mitigate calibration issues. This surgery adds a factor to the loss's partial derivative, scaling the gradient's magnitude linearly with the prediction error. In empirical evaluations across 2D and 3D medical segmentation tasks, we demonstrate the effectiveness of this intervention while maintaining high prediction accuracy when used in conjunction with any region-based loss function.