# Vision-Language Enhanced Foundation Model for Semi-Supervised Medical Image Segmentation

arXiv: https://arxiv.org/abs/2511.19759 · HTML: https://mlsanity.com/paper/2511.19759/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 50 of its 72 references that are in the map.
- First posted on arXiv 24 Nov 2025.

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision](https://mlsanity.com/paper/2106.01226/) (2021)
- [Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation](https://mlsanity.com/paper/1907.07034/) (2019)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation](https://mlsanity.com/paper/1809.10486/) (2018)
- [Interpolation Consistency Training for Semi-Supervised Learning](https://mlsanity.com/paper/1903.03825/) (2019)
- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography](https://mlsanity.com/paper/1908.06948/) (2019)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2208.09910/) (2022)
- [Adversarial Learning for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/1802.07934/) (2018)

## Abstract (arXiv)

Semi-supervised learning (SSL) has emerged as an efficient paradigm for medical image segmentation, reducing the reliance on extensive expert annotations. Vision-language models (VLMs) have demonstrated strong generalization and few-shot capabilities across diverse visual domains. In this work, we integrate a VLM into a semi-supervised medical image segmentation model by adding a Vision-Language Enhanced Semi-supervised Segmentation Assistant (VESSA) that incorporates foundation-level visual-semantic understanding into SSL frameworks. Our approach consists of two stages. In Stage 1, the VLM-enhanced segmentation foundation model VESSA is trained as a reference-guided segmentation assistant using a template bank containing gold-standard exemplars, simulating learning from limited labeled data. Given an input-template pair, VESSA performs visual feature matching to extract representative semantic and spatial cues from exemplar segmentations, generating structured prompts for a Segment Anything Model (SAM)-inspired mask decoder to produce segmentation masks. In Stage 2, VESSA is integrated into an SSL framework as a plug-and-play teacher, providing template-guided pseudo-labels that complement the task-specific student model and strengthen supervision under scarce annotations. Extensive experiments across multiple segmentation datasets and domains show that VESSA-augmented SSL significantly enhances segmentation accuracy, outperforming state-of-the-art baselines under extremely limited annotation conditions.