# Confidence-Calibrating Regularization for Robust Brain MRI Segmentation Under Domain Shift

arXiv: https://arxiv.org/abs/2509.23176 · HTML: https://mlsanity.com/paper/2509.23176/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Builds on (most cited first, 22 of 22)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Unsupervised domain adaptation in brain lesion segmentation with adversarial networks](https://mlsanity.com/paper/1612.08894/) (2016)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [Fisher SAM: Information Geometry and Sharpness Aware Minimisation](https://mlsanity.com/paper/2206.04920/) (2022)
- [CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training](https://mlsanity.com/paper/2310.13292/) (2023)
- [Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors](https://mlsanity.com/paper/1712.09376/) (2017)
- [Assessing Reliability and Challenges of Uncertainty Estimations for Medical Image Segmentation](https://mlsanity.com/paper/1907.03338/) (2019)
- [Segment Anything Model for Medical Images?](https://mlsanity.com/paper/2304.14660/) (2023)
- [Dual-Reference Source-Free Active Domain Adaptation for Nasopharyngeal Carcinoma Tumor Segmentation across Multiple Hospitals](https://mlsanity.com/paper/2309.13401/) (2023)
- [On the Variance of the Fisher Information for Deep Learning](https://mlsanity.com/paper/2107.04205/) (2021)
- [Calibrating the Dice loss to handle neural network overconfidence for biomedical image segmentation](https://mlsanity.com/paper/2111.00528/) (2021)
- [Threshold-Consistent Margin Loss for Open-World Deep Metric Learning](https://mlsanity.com/paper/2307.04047/) (2023)
- [Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models](https://mlsanity.com/paper/2502.07847/) (2025)

## Abstract (arXiv)

The Segment Anything Model (SAM) exhibits strong zero-shot performance on natural images but suffers from domain shift and overconfidence when applied to medical volumes. We propose CalSAM, a lightweight adaptation framework that (i) reduces encoder sensitivity to domain shift via a Feature Fisher Information Penalty (FIP) computed on 3D feature maps and (ii) penalizes overconfident voxel-wise errors through a Confidence Misalignment Penalty (CMP). The combined loss, (LCalSAM) fine-tunes only the mask decoder while keeping SAM's encoders frozen. On cross-center and scanner-shift evaluations, CalSAM substantially improves accuracy and calibration: e.g., on the BraTS scanner split (Siemens→GE) CalSAM shows a +7.4% relative improvement in DSC (80.1% vs. 74.6%), a -26.9% reduction in HD95 (4.6 mm vs. 6.3 mm), and a -39.5% reduction in ECE (5.2% vs. 8.6%). On ATLAS-C (motion corruptions), CalSAM achieves a +5.3% relative improvement in DSC (75.9%) and a -32.6% reduction in ECE (5.8%). Ablations show FIP and CMP contribute complementary gains (p<0.01), and the Fisher penalty incurs a modest ~15% training-time overhead. CalSAM therefore delivers improved domain generalization and better-calibrated uncertainty estimates for brain MRI segmentation, while retaining the computational benefits of freezing SAM's encoder.