# Can language-guided unsupervised adaptation improve medical image classification using unpaired images and texts?

arXiv: https://arxiv.org/abs/2409.02729 · HTML: https://mlsanity.com/paper/2409.02729/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 7 of 7)

- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [Are Natural Domain Foundation Models Useful for Medical Image Classification?](https://mlsanity.com/paper/2310.19522/) (2023)
- [LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections](https://mlsanity.com/paper/2305.18287/) (2023)
- [The Entropy Enigma: Success and Failure of Entropy Minimization](https://mlsanity.com/paper/2405.05012/) (2024)
- [Freeze the backbones: A Parameter-Efficient Contrastive Approach to Robust Medical Vision-Language Pre-training](https://mlsanity.com/paper/2401.01179/) (2024)
- [UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Prompts](https://mlsanity.com/paper/2312.11171/) (2023)

## Built on by (most cited first, 2 of 2)

- [Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models](https://mlsanity.com/paper/2605.08787/) (2026)
- [Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation under Shift](https://mlsanity.com/paper/2509.09397/) (2025)

## Abstract (arXiv)

In medical image classification, supervised learning is challenging due to the scarcity of labeled medical images. To address this, we leverage the visual-textual alignment within Vision-Language Models (VLMs) to enable unsupervised learning of a medical image classifier. In this work, we propose underlineMedical underlineUnsupervised underlineAdaptation (textttMedUnA) of VLMs, where the LLM-generated descriptions for each class are encoded into text embeddings and matched with class labels via a cross-modal adapter. This adapter attaches to a visual encoder of textttMedCLIP and aligns the visual embeddings through unsupervised learning, driven by a contrastive entropy-based loss and prompt tuning. Thereby, improving performance in scenarios where textual information is more abundant than labeled images, particularly in the healthcare domain. Unlike traditional VLMs, textttMedUnA uses unpaired images and text for learning representations and enhances the potential of VLMs beyond traditional constraints. We evaluate the performance on three chest X-ray datasets and two multi-class datasets (diabetic retinopathy and skin lesions), showing significant accuracy gains over the zero-shot baseline. Our code is available at https://github.com/rumaima/meduna.