# Uncertainty-Aware Vision-Language Segmentation for Medical Imaging

arXiv: https://arxiv.org/abs/2602.14498 · HTML: https://mlsanity.com/paper/2602.14498/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 16 Feb 2026.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer](https://mlsanity.com/paper/2109.04335/) (2021)
- [Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining](https://mlsanity.com/paper/2402.03302/) (2024)
- [LViT: Language meets Vision Transformer in Medical Image Segmentation](https://mlsanity.com/paper/2206.14718/) (2022)
- [MosMedData: Chest CT Scans With COVID-19 Related Findings Dataset](https://mlsanity.com/paper/2005.06465/) (2020)
- [Beyond One-to-One: Rethinking the Referring Image Segmentation](https://mlsanity.com/paper/2308.13853/) (2023)
- [TGANet: Text-guided attention for improved polyp segmentation](https://mlsanity.com/paper/2205.04280/) (2022)
- [OSegNet: Operational Segmentation Network for COVID-19 Detection using Chest X-ray Images](https://mlsanity.com/paper/2202.10185/) (2022)
- [Bayesian QuickNAT: Model Uncertainty in Deep Whole-Brain Segmentation for Structure-wise Quality Control](https://mlsanity.com/paper/1811.09800/) (2018)
- [Uncertainty guided semi-supervised segmentation of retinal layers in OCT images](https://mlsanity.com/paper/2103.02083/) (2021)
- [Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models](https://mlsanity.com/paper/2308.07706/) (2023)
- [Cross-Modal Conditioned Reconstruction for Language-guided Medical Image Segmentation](https://mlsanity.com/paper/2404.02845/) (2024)
- [MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training](https://mlsanity.com/paper/2411.15576/) (2024)
- [Language-guided Medical Image Segmentation with Target-informed Multi-level Contrastive Alignments](https://mlsanity.com/paper/2412.13533/) (2024)

## Built on by (most cited first, 5 of 5)

- [UNITY: Attention Flow Networks for Adaptive Conditioning in Diffusion](https://mlsanity.com/paper/2606.20971/) (2026)
- [SALD: Self-Referenced Advantage Learning for Diffusion Models](https://mlsanity.com/paper/2610.01496/) (2026)
- [Prototype-Rectified Iterative Self-supervised Manifold Denoising under Severe Acoustic Shift](https://mlsanity.com/paper/2608.15037/) (2026)
- [Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations](https://mlsanity.com/paper/2610.14944/) (2026)
- [Function-Space Temporal Self-Distillation for Policy Optimization](https://mlsanity.com/paper/2610.19687/) (2026)

## Abstract (arXiv)

We introduce a novel uncertainty-aware multimodal segmentation framework that leverages both radiological images and associated clinical text for precise medical diagnosis. We propose a Modality Decoding Attention Block (MoDAB) with a lightweight State Space Mixer (SSMix) to enable efficient cross-modal fusion and long-range dependency modelling. To guide learning under ambiguity, we propose the Spectral-Entropic Uncertainty (SEU) Loss, which jointly captures spatial overlap, spectral consistency, and predictive uncertainty in a unified objective. In complex clinical circumstances with poor image quality, this formulation improves model reliability. Extensive experiments on various publicly available medical datasets, QATA-COVID19, MosMed++, and Kvasir-SEG, demonstrate that our method achieves superior segmentation performance while being significantly more computationally efficient than existing State-of-the-Art (SoTA) approaches. Our results highlight the importance of incorporating uncertainty modelling and structured modality alignment in vision-language medical segmentation tasks. Code: https://github.com/arya-domain/UA-VLS