# Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models

arXiv: https://arxiv.org/abs/2604.10963 · HTML: https://mlsanity.com/paper/2604.10963/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 43 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Builds on (most cited first, 22 of 22)

- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT](https://mlsanity.com/paper/2307.01984/) (2023)
- [MedSAM2: Segment Anything in 3D Medical Images and Videos](https://mlsanity.com/paper/2504.03600/) (2025)
- [SegVol: Universal and Interactive Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2311.13385/) (2023)
- [Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks](https://mlsanity.com/paper/2402.19460/) (2024)
- [An automatic multi-tissue human fetal brain segmentation benchmark using the Fetal Tissue Annotation Dataset](https://mlsanity.com/paper/2010.15526/) (2020)
- [Uncertainty Estimation by Fisher Information-based Evidential Deep Learning](https://mlsanity.com/paper/2303.02045/) (2023)
- [A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods](https://mlsanity.com/paper/2310.06873/) (2023)
- [Delving into Semantic Scale Imbalance](https://mlsanity.com/paper/2212.14613/) (2022)
- [URL: A Representation Learning Benchmark for Transferable Uncertainty Estimates](https://mlsanity.com/paper/2307.03810/) (2023)
- [Learning Sample Difficulty from Pre-trained Models for Reliable Prediction](https://mlsanity.com/paper/2304.10127/) (2023)
- [One step closer to unbiased aleatoric uncertainty estimation](https://mlsanity.com/paper/2312.10469/) (2023)
- [Predicting and Enhancing the Fairness of DNNs with the Curvature of Perceptual Manifolds](https://mlsanity.com/paper/2303.12307/) (2023)
- [Angular Gap: Reducing the Uncertainty of Image Difficulty through Model Calibration](https://mlsanity.com/paper/2207.08525/) (2022)
- [Unveiling and Mitigating Generalized Biases of DNNs through the Intrinsic Dimensions of Perceptual Manifolds](https://mlsanity.com/paper/2404.13859/) (2024)
- [Rethinking Epistemic and Aleatoric Uncertainty for Active Open-Set Annotation: An Energy-Based Approach](https://mlsanity.com/paper/2502.19691/) (2025)
- [Exploring Aleatoric Uncertainty in Object Detection via Vision Foundation Models](https://mlsanity.com/paper/2411.17767/) (2024)

## Built on by (most cited first, 1 of 1)

- [G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification](https://mlsanity.com/paper/2608.26744/) (2026)

## Abstract (arXiv)

Medical image segmentation supports clinical workflows by precisely delineating anatomical structures and lesions. However, medical image datasets medical image datasets suffer from acquisition noise and annotation ambiguity, causing pervasive data uncertainty that substantially undermines model robustness. Existing research focuses primarily on model architectural improvements and predictive reliability estimation, while systematic exploration of the intrinsic data uncertainty remains insufficient. To address this gap, this work proposes leveraging the universal representation capabilities of visual foundation models to estimate inherent data uncertainty. Specifically, we analyze the feature diversity of the model's decoded representations and quantify their singular value energy to define the semantic perception scale for each class, thereby measuring sample difficulty and aleatoric uncertainty. Based on this foundation, we design two uncertainty-driven application strategies: (1) the aleatoric uncertainty-aware data filtering mechanism to eliminate potentially noisy samples and enhance model learning quality; (2) the dynamic uncertainty-aware optimization strategy that adaptively adjusts class-specific loss weights during training based on the semantic perception scale, combined with a label denoising mechanism to improve training stability. Experimental results on five public datasets encompassing CT and MRI modalities and involving multi-organ and tumor segmentation tasks demonstrate that our method achieves significant and robust performance improvements across various mainstream network architectures, revealing the broad application potential of aleatoric uncertainty in medical image understanding and segmentation tasks.