# UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities

arXiv: https://arxiv.org/abs/2412.10372 · HTML: https://mlsanity.com/paper/2412.10372/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 13 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Built on by (most cited first, 30 of 46)

- [Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner](https://mlsanity.com/paper/2505.11404/) (2025)
- [FetalCLIP: A Visual-Language Foundation Model for Fetal Ultrasound Image Analysis](https://mlsanity.com/paper/2502.14807/) (2025)
- [On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?](https://mlsanity.com/paper/2505.15425/) (2025)
- [MedMoE: Modality-Specialized Mixture of Experts for Medical Vision-Language Understanding](https://mlsanity.com/paper/2506.08356/) (2025)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [Adapting Vision-Language Foundation Model for Next Generation Medical Ultrasound Image Analysis](https://mlsanity.com/paper/2506.08849/) (2025)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [Delving into Out-of-Distribution Detection with Medical Vision-Language Models](https://mlsanity.com/paper/2503.01020/) (2025)
- [MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2602.20423/) (2026)
- [Ultrasound-CLIP: Semantic-Aware Contrastive Pre-training for Ultrasound Image-Text Understanding](https://mlsanity.com/paper/2604.01749/) (2026)
- [Multimodal Medical Image Binding via Shared Text Embeddings](https://mlsanity.com/paper/2506.18072/) (2025)
- [Boosting Medical Visual Understanding From Multi-Granular Language Learning](https://mlsanity.com/paper/2511.15943/) (2025)
- [Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2](https://mlsanity.com/paper/2511.05509/) (2025)
- [Semantic search for 100M+ galaxy images using AI-generated captions](https://mlsanity.com/paper/2512.11982/) (2025)
- [Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models](https://mlsanity.com/paper/2512.13072/) (2025)
- [Enhancing Medical Large Vision-Language Models via Alignment Distillation](https://mlsanity.com/paper/2512.18554/) (2025)
- [Natural Language-Driven Global Mapping of Martian Landforms](https://mlsanity.com/paper/2601.15949/) (2026)
- [Sparse Spectral LoRA: Routed Experts for Medical VLMs](https://mlsanity.com/paper/2604.01310/) (2026)
- [Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift](https://mlsanity.com/paper/2607.10358/) (2026)
- [Leveraging Textual Anatomical Knowledge for Class-Imbalanced Semi-Supervised Multi-Organ Segmentation](https://mlsanity.com/paper/2501.13470/) (2025)
- [Multimodal, Multi-Disease Medical Imaging Foundation Model (MerMED-FM)](https://mlsanity.com/paper/2507.00185/) (2025)
- [CLAPS: A CLIP-Unified Auto-Prompt Segmentation for Multi-Modal Retinal Imaging](https://mlsanity.com/paper/2509.08618/) (2025)
- [UOPSL: Unpaired OCT Predilection Sites Learning for Fundus Image Diagnosis Augmentation](https://mlsanity.com/paper/2509.08624/) (2025)
- [Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysis](https://mlsanity.com/paper/2601.14774/) (2026)
- [DoubleTake: Contrastive Reasoning for Faithful Decision-Making in Medical Imaging](https://mlsanity.com/paper/2602.02894/) (2026)
- [XAI-CLIP: ROI-Guided Perturbation Framework for Explainable Medical Image Segmentation in Multimodal Vision-Language Models](https://mlsanity.com/paper/2602.07017/) (2026)
- [NAST: Improving Negation Handling in Medical Vision-Language Models through Negation-Aware Selective Training](https://mlsanity.com/paper/2602.12498/) (2026)
- [DARK: Diagonal-Anchored Repulsive Knowledge Distillation for Vision-Language Models under Extreme Compression](https://mlsanity.com/paper/2603.05421/) (2026)
- [DBMF: A Dual-Branch Multimodal Framework for Out-of-Distribution Detection](https://mlsanity.com/paper/2604.08261/) (2026)
- [M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model](https://mlsanity.com/paper/2604.08936/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) trained via contrastive learning have achieved notable success in natural image tasks. However, their application in the medical domain remains limited due to the scarcity of openly accessible, large-scale medical image-text datasets. Existing medical VLMs either train on closed-source proprietary or relatively small open-source datasets that do not generalize well. Similarly, most models remain specific to a single or limited number of medical imaging domains, again restricting their applicability to other modalities. To address this gap, we introduce UniMed, a large-scale, open-source multi-modal medical dataset comprising over 5.3 million image-text pairs across six diverse imaging modalities: X-ray, CT, MRI, Ultrasound, Pathology, and Fundus. UniMed is developed using a data-collection framework that leverages Large Language Models (LLMs) to transform modality-specific classification datasets into image-text formats while incorporating existing image-text data from the medical domain, facilitating scalable VLM pretraining. Using UniMed, we trained UniMed-CLIP, a unified VLM for six modalities that significantly outperforms existing generalist VLMs and matches modality-specific medical VLMs, achieving notable gains in zero-shot evaluations. For instance, UniMed-CLIP improves over BiomedCLIP (trained on proprietary data) by an absolute gain of +12.61, averaged over 21 datasets, while using 3x less training data. To facilitate future research, we release UniMed dataset, training codes, and models at https://github.com/mbzuai-oryx/UniMed-CLIP.