# UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model

arXiv: https://arxiv.org/abs/2505.05049 · HTML: https://mlsanity.com/paper/2505.05049/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 8 May 2025.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference](https://mlsanity.com/paper/1506.02158/) (2015)
- [Bayesian Neural Networks: An Introduction and Survey](https://mlsanity.com/paper/2006.12024/) (2020)
- [AutoMix: Automatically Mixing Language Models](https://mlsanity.com/paper/2310.12963/) (2023)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- [Segment anything, from space?](https://mlsanity.com/paper/2304.13000/) (2023)
- [One-Prompt to Segment All Medical Images](https://mlsanity.com/paper/2305.10300/) (2023)
- [Biomedical SAM 2: Segment Anything in Biomedical Images and Videos](https://mlsanity.com/paper/2408.03286/) (2024)
- [Blind Knowledge Distillation for Robust Image Classification](https://mlsanity.com/paper/2211.11355/) (2022)
- [HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization](https://mlsanity.com/paper/2308.07163/) (2023)
- [Compensation Learning in Semantic Segmentation](https://mlsanity.com/paper/2304.13428/) (2023)
- [Segment-Anything Models Achieve Zero-shot Robustness in Autonomous Driving](https://mlsanity.com/paper/2408.09839/) (2024)
- [Multi-Flow: Multi-View-Enriched Normalizing Flows for Industrial Anomaly Detection](https://mlsanity.com/paper/2504.03306/) (2025)
- [Strike the Balance: On-the-Fly Uncertainty based User Interactions for Long-Term Video Object Segmentation](https://mlsanity.com/paper/2408.00169/) (2024)
- [Utilizing Uncertainty in 2D Pose Detectors for Probabilistic 3D Human Mesh Recovery](https://mlsanity.com/paper/2411.16289/) (2024)
- [Towards Collective Intelligence: Uncertainty-aware SAM Adaptation for Ambiguous Medical Image Segmentation](https://mlsanity.com/paper/2403.10931/) (2024)
- [Cell Tracking according to Biological Needs -- Strong Mitosis-aware Multi-Hypothesis Tracker with Aleatoric Uncertainty](https://mlsanity.com/paper/2403.15011/) (2024)
- [Parameter-efficient Bayesian Neural Networks for Uncertainty-aware Depth Estimation](https://mlsanity.com/paper/2409.17085/) (2024)

## Built on by (most cited first, 4 of 4)

- [Pruning by Block Benefit: Exploring the Properties of Vision Transformer Blocks during Domain Adaptation](https://mlsanity.com/paper/2506.23675/) (2025)
- [Towards Integrating Uncertainty for Domain-Agnostic Segmentation](https://mlsanity.com/paper/2512.23427/) (2025)
- [Disentangling Prompt Dependence to Evaluate Segmentation Reliability in Gynecological MRI](https://mlsanity.com/paper/2603.13369/) (2026)
- [DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classification](https://mlsanity.com/paper/2604.07166/) (2026)

## Abstract (arXiv)

The introduction of the Segment Anything Model (SAM) has paved the way for numerous semantic segmentation applications. For several tasks, quantifying the uncertainty of SAM is of particular interest. However, the ambiguous nature of the class-agnostic foundation model SAM challenges current uncertainty quantification (UQ) approaches. This paper presents a theoretically motivated uncertainty quantification model based on a Bayesian entropy formulation jointly respecting aleatoric, epistemic, and the newly introduced task uncertainty. We use this formulation to train USAM, a lightweight post-hoc UQ method. Our model traces the root of uncertainty back to under-parameterised models, insufficient prompts or image ambiguities. Our proposed deterministic USAM demonstrates superior predictive capabilities on the SA-V, MOSE, ADE20k, DAVIS, and COCO datasets, offering a computationally cheap and easy-to-use UQ alternative that can support user-prompting, enhance semi-supervised pipelines, or balance the tradeoff between accuracy and cost efficiency.