mlsanityData to 7 Oct 2026

MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

arXiv 2602.20423 ↗First posted 23 Feb 2026cs.CV cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 82 references that are in the map. First posted on arXiv 23 Feb 2026.

Builds on

59 of 82 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025VLSM-Ensemble: Ensembling CLIP-based Vision-Language Models for Enhanced Medical Image Segmentation5 Sep
  2. 2024UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities13 Dec
  3. 2024BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models21 Nov
  4. 2024VLSM-Adapter: Finetuning Vision-Language Segmentation Efficiently with Lightweight Blocks10 May
  5. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models31 Mar
  6. 2023Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models15 Aug

Other references · 53

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  7. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  8. 2018UNet++: A Nested U-Net Architecture for Medical Image Segmentationcited by 1,518
  9. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  10. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,420
  11. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
  12. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 954
38 more
  1. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  2. 2019Kvasir-SEG: A Segmented Polyp Datasetcited by 588
  3. 2021Extract Free Dense Labels from CLIPcited by 580
  4. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  5. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  6. 2021Image Segmentation Using Text and Image Promptscited by 509
  7. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  8. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  9. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  10. 2020CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentationcited by 307
  11. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  12. 2016A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Imagescited by 220
  13. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  14. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  15. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  16. 2019Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentationcited by 162
  17. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  18. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  19. 2022HiFormer: Hierarchical Multi-scale Representations Using Transformers for Medical Image Segmentationcited by 94
  20. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  21. 2024Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Imagescited by 69
  22. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 63
  23. 2025Your ViT is Secretly an Image Segmentation Modelcited by 61
  24. 2021NeoUNet: Towards accurate colon polyp segmentation and neoplasm detectioncited by 50
  25. 2022OSegNet: Operational Segmentation Network for COVID-19 Detection using Chest X-ray Imagescited by 47
  26. 2023ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Imagecited by 47
  27. 2023CLIP in Medical Imaging: A Surveycited by 40
  28. 2024MedCLIP-SAM: Bridging Text and Image Towards Universal Medical Image Segmentationcited by 37
  29. 2024MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentationcited by 37
  30. 2024Robust Calibration of Large Vision-Language Adapterscited by 28
  31. 2023Domain-Controlled Prompt Learningcited by 22
  32. 2022Efficient Bayesian Uncertainty Estimation for nnU-Netcited by 19
  33. 2023Mitigating Transformer Overconfidence via Lipschitz Regularizationcited by 15
  34. 2025CausalCLIPSeg: Unlocking CLIP's Potential in Referring Medical Image Segmentation with Causal Interventioncited by 15
  35. 2025Show and Segment: Universal Medical Image Segmentation via In-Context Learningcited by 15
  36. 2023Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibrationcited by 14
  37. 2025BRISC: Annotated Dataset for Brain Tumor Segmentation and Classificationcited by 14
  38. 2024MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidancecited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuningcited by 0
  2. 2026DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentationcited by 0
  3. 2026Characterizing Text Branch Sensitivity in Medical Vision-Language Segmentation via Evidence Decouplingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026BiCLIP: Bidirectional and Consistent Language-Image Processing for Robust Medical Image Segmentation
  2. 2026MedPlex: Deep Vision-Language Co-Adaptation for Clinically Grounded Medical Segmentation
  3. 2026XAI-CLIP: ROI-Guided Perturbation Framework for Explainable Medical Image Segmentation in Multimodal Vision-Language Models
  4. 2025BiPVL-Seg: Bidirectional Progressive Vision-Language Fusion with Global-Local Alignment for Medical Image Segmentation
  5. 2026SAM Meets VLM: Parameter-Decoupled Full-Parameter Training for Unified Medical Reasoning and Segmentation
  6. 2026SegBanana: Steering Unified Multimodal Models into Medical Segmenters
  7. 2026SegMoTE: Token-Level Mixture of Experts for Medical Image Segmentation
  8. 2026MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration