# Trust your neighbours: Penalty-based constraints for model calibration

arXiv: https://arxiv.org/abs/2303.06268 · HTML: https://mlsanity.com/paper/2303.06268/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 11 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge](https://mlsanity.com/paper/1811.02629/) (2018)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Local Temperature Scaling for Probability Calibration](https://mlsanity.com/paper/2008.05105/) (2020)
- [The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration](https://mlsanity.com/paper/2111.15430/) (2021)
- [Post-hoc Uncertainty Calibration for Domain Drift Scenarios](https://mlsanity.com/paper/2012.10988/) (2020)
- [Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations](https://mlsanity.com/paper/2104.05788/) (2021)

## Built on by (most cited first, 12 of 12)

- [Robust Calibration of Large Vision-Language Adapters](https://mlsanity.com/paper/2407.13588/) (2024)
- [Neighbor-Aware Calibration of Segmentation Networks with Penalty-Based Constraints](https://mlsanity.com/paper/2401.14487/) (2024)
- [Average Calibration Error: A Differentiable Loss for Improved Reliability in Image Segmentation](https://mlsanity.com/paper/2403.06759/) (2024)
- [Boundary-weighted logit consistency improves calibration of segmentation networks](https://mlsanity.com/paper/2307.08163/) (2023)
- [Average Calibration Losses for Reliable Uncertainty in Medical Image Segmentation](https://mlsanity.com/paper/2506.03942/) (2025)
- [Class and Region-Adaptive Constraints for Network Calibration](https://mlsanity.com/paper/2403.12364/) (2024)
- [Do not trust what you trust: Miscalibration in Semi-supervised Learning](https://mlsanity.com/paper/2403.15567/) (2024)
- [We Care Each Pixel: Calibrating on Medical Segmentation Model](https://mlsanity.com/paper/2503.05107/) (2025)
- [h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective](https://mlsanity.com/paper/2506.17968/) (2025)
- [Quantile Adaptive Temperature Scaling for Confidence Calibration](https://mlsanity.com/paper/2606.21749/) (2026)
- [Improving Uncertainty-Error Correspondence in Deep Bayesian Medical Image Segmentation](https://mlsanity.com/paper/2409.03470/) (2024)
- [Beyond scalar losses: calibrating segmentation models via gradient vector field surgery](https://mlsanity.com/paper/2607.14338/) (2026)

## Abstract (arXiv)

Ensuring reliable confidence scores from deep networks is of pivotal importance in critical decision-making systems, notably in the medical domain. While recent literature on calibrating deep segmentation networks has led to significant progress, their uncertainty is usually modeled by leveraging the information of individual pixels, which disregards the local structure of the object of interest. In particular, only the recent Spatially Varying Label Smoothing (SVLS) approach addresses this issue by softening the pixel label assignments with a discrete spatial Gaussian kernel. In this work, we first present a constrained optimization perspective of SVLS and demonstrate that it enforces an implicit constraint on soft class proportions of surrounding pixels. Furthermore, our analysis shows that SVLS lacks a mechanism to balance the contribution of the constraint with the primary objective, potentially hindering the optimization process. Based on these observations, we propose a principled and simple solution based on equality constraints on the logit values, which enables to control explicitly both the enforced constraint and the weight of the penalty, offering more flexibility. Comprehensive experiments on a variety of well-known segmentation benchmarks demonstrate the superior performance of the proposed approach.