mlsanityData to 7 Oct 2026

Characterizing Text Branch Sensitivity in Medical Vision-Language Segmentation via Evidence Decoupling

arXiv 2609.02663 ↗First posted 2 Sep 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Pretrained vision-language models (VLMs) have shown promising performance in medical image segmentation by incorporating clinical text. However, it remains unclear how much textual information actually contributes to pixel-level predictions. In this work, we systematically investigate the role of text in multimodal medical image segmentation. We first analyze several commonly used fusion strategies and find that segmentation performance is largely insensitive to the choice of fusion module. To further understand modality interactions, we propose an Evidence Decoupling Decoder (EDD) based on evidential deep learning and deep supervision. EDD serves as an internal representation analysis tool that decomposes image evidence and text-modulated evidence throughout the decoding process while maintaining competitive segmentation performance. Experimental results show that the sensitivity to text perturbation varies substantially across datasets. On BUSI and BTMRI, removing text causes catastrophic performance drops, indicating strong model reliance on textual input. On ISIC and Kvasir-SEG, text exerts relatively marginal influence. We further find that text affects predictions mainly through global semantic modulation rather than independent spatial localization, and that the specific semantic components driving text sensitivity differ across datasets. These findings provide a deeper understanding of modality interaction in multimodal medical image segmentation and offer practical insights for future model design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 37 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

33 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation23 Feb
  2. 2024UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities13 Dec

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  6. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  7. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  8. 2018UNet++: A Nested U-Net Architecture for Medical Image Segmentationcited by 1,518
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
  11. 2019Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)cited by 657
  12. 2019Kvasir-SEG: A Segmented Polyp Datasetcited by 588
19 more
  1. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  2. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  3. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  4. 2014Deeply-Supervised Netscited by 364
  5. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  6. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  7. 2022TGANet: Text-guided attention for improved polyp segmentationcited by 59
  8. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 55
  9. 2024Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learningcited by 25
  10. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  11. 2022Region-Based Evidential Deep Learning to Quantify Uncertainty and Improve Robustness of Brain Tumor Segmentationcited by 20
  12. 2022A Dempster-Shafer approach to trustworthy AI with application to fetal brain MRI segmentationcited by 11
  13. 2025See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Biascited by 9
  14. 2024MLIP: Medical Language-Image Pre-training with Masked Local Representation Learningcited by 7
  15. 2024Uncertainty-aware Evidential Fusion-based Learning for Semi-supervised Medical Image Segmentationcited by 6
  16. 2024Language-guided Medical Image Segmentation with Target-informed Multi-level Contrastive Alignmentscited by 6
  17. 2025Uncertainty-Supervised Interpretable and Robust Evidential Segmentationcited by 3
  18. 2023Deep evidential fusion with uncertainty quantification and contextual discounting for multimodal medical image segmentationcited by 2
  19. 2025Progressive Uncertainty-Guided Evidential U-KAN for Trustworthy Medical Image Segmentationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SAM Meets VLM: Parameter-Decoupled Full-Parameter Training for Unified Medical Reasoning and Segmentation
  2. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models
  3. 2026MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation
  4. 2025CausalCLIPSeg: Unlocking CLIP's Potential in Referring Medical Image Segmentation with Causal Intervention
  5. 2025Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation
  6. 2025BiPVL-Seg: Bidirectional Progressive Vision-Language Fusion with Global-Local Alignment for Medical Image Segmentation
  7. 2025Causal-SAM-LLM: Large Language Models as Causal Reasoners for Robust Medical Segmentation
  8. 2026XAI-CLIP: ROI-Guided Perturbation Framework for Explainable Medical Image Segmentation in Multimodal Vision-Language Models