# RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports

arXiv: https://arxiv.org/abs/2405.14137 · HTML: https://mlsanity.com/paper/2405.14137/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 23 May 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents](https://mlsanity.com/paper/2303.07240/) (2023)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://mlsanity.com/paper/2211.01335/) (2022)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)
- [ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Data](https://mlsanity.com/paper/2310.05242/) (2023)
- [Exploring the Transfer Learning Capabilities of CLIP in Domain Generalization for Diabetic Retinopathy](https://mlsanity.com/paper/2308.14212/) (2023)

## Built on by (most cited first, 30 of 30)

- [Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions](https://mlsanity.com/paper/2412.02621/) (2024)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [ViLReF: An Expert Knowledge Enabled Vision-Language Retinal Foundation Model](https://mlsanity.com/paper/2408.10894/) (2024)
- [Applications of Large Models in Medicine](https://mlsanity.com/paper/2502.17132/) (2025)
- [MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining](https://mlsanity.com/paper/2501.15798/) (2025)
- [RetinaLogos: Fine-Grained Synthesis of High-Resolution Retinal Images Through Captions](https://mlsanity.com/paper/2505.12887/) (2025)
- [Continual Retinal Vision-Language Pre-training upon Incremental Imaging Modalities](https://mlsanity.com/paper/2506.19320/) (2025)
- [A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Models](https://mlsanity.com/paper/2508.03734/) (2025)
- [Fine-tuning Vision Language Models with Graph-based Knowledge for Explainable Medical Image Analysis](https://mlsanity.com/paper/2503.09808/) (2025)
- [PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning](https://mlsanity.com/paper/2505.12233/) (2025)
- [Boosting Medical Visual Understanding From Multi-Granular Language Learning](https://mlsanity.com/paper/2511.15943/) (2025)
- [An Explainable Transformer Model for Alzheimer's Disease Detection Using Retinal Imaging](https://mlsanity.com/paper/2507.04259/) (2025)
- [PSScreen: Partially Supervised Multiple Retinal Disease Screening](https://mlsanity.com/paper/2508.10549/) (2025)
- [Native Intelligence Emerges from Large-Scale Clinical Practice: A Retinal Foundation Model with Deployment Efficiency](https://mlsanity.com/paper/2512.14499/) (2025)
- [REVEAL: Multimodal Vision-Language Alignment of Retinal Morphometry and Clinical Risks for Incident AD and Dementia Prediction](https://mlsanity.com/paper/2604.18757/) (2026)
- [OphIn-500K: Curating Web-Scale Visual Instructions for Scaling Ophthalmic Multimodal Large Language Models](https://mlsanity.com/paper/2605.27916/) (2026)
- [CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignment](https://mlsanity.com/paper/2508.06434/) (2025)
- [UOPSL: Unpaired OCT Predilection Sites Learning for Fundus Image Diagnosis Augmentation](https://mlsanity.com/paper/2509.08624/) (2025)
- [Evaluating Fundus-Specific Foundation Models for Diabetic Macular Edema Detection](https://mlsanity.com/paper/2510.07277/) (2025)
- [PSScreen V2: Partially Supervised Multiple Retinal Disease Screening](https://mlsanity.com/paper/2510.22589/) (2025)
- [PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregation](https://mlsanity.com/paper/2602.23297/) (2026)
- [Towards Interpretable Foundation Models for Retinal Fundus Images](https://mlsanity.com/paper/2603.18846/) (2026)
- [Bridging Restoration and Diagnosis: A Comprehensive Benchmark for Retinal Fundus Enhancement](https://mlsanity.com/paper/2604.03806/) (2026)
- [Scientific Domain Knowledge Improves Vision-Language Fundus Models](https://mlsanity.com/paper/2605.02720/) (2026)
- [Cross-Modal Semantic-Enhanced Diffusion Framework for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2605.09242/) (2026)
- [A Cascaded Edge-Cloud Architecture for Automated Diabetic Retinopathy Screening](https://mlsanity.com/paper/2605.14108/) (2026)
- [REVEAL++: Differentiable Phenotypic Grouping for Vision-Language Retinal Modeling of Alzheimer's Disease Risk](https://mlsanity.com/paper/2606.19522/) (2026)
- [Adaptive Conformal Redistribution for Inter-class Transitional Uncertainty in Medical Image Classification](https://mlsanity.com/paper/2609.13303/) (2026)
- [FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs](https://mlsanity.com/paper/2609.33158/) (2026)
- [Exploring the Latent Structure of AMD Severity Using Retinal Foundation Models](https://mlsanity.com/paper/2610.14138/) (2026)

## Abstract (arXiv)

The Vision-Language Foundation model is increasingly investigated in the fields of computer vision and natural language processing, yet its exploration in ophthalmology and broader medical applications remains limited. The challenge is the lack of labeled data for the training of foundation model. To handle this issue, a CLIP-style retinal image foundation model is developed in this paper. Our foundation model, RET-CLIP, is specifically trained on a dataset of 193,865 patients to extract general features of color fundus photographs (CFPs), employing a tripartite optimization strategy to focus on left eye, right eye, and patient level to reflect real-world clinical scenarios. Extensive experiments demonstrate that RET-CLIP outperforms existing benchmarks across eight diverse datasets spanning four critical diagnostic categories: diabetic retinopathy, glaucoma, multiple disease diagnosis, and multi-label classification of multiple diseases, which demonstrate the performance and generality of our foundation model. The sourse code and pre-trained model are available at https://github.com/sStonemason/RET-CLIP.