mlsanityData to 7 Oct 2026

RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports

arXiv 2405.14137 ↗First posted 23 May 2024cs.CVcited by 29

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The Vision-Language Foundation model is increasingly investigated in the fields of computer vision and natural language processing, yet its exploration in ophthalmology and broader medical applications remains limited. The challenge is the lack of labeled data for the training of foundation model. To handle this issue, a CLIP-style retinal image foundation model is developed in this paper. Our foundation model, RET-CLIP, is specifically trained on a dataset of 193,865 patients to extract general features of color fundus photographs (CFPs), employing a tripartite optimization strategy to focus on left eye, right eye, and patient level to reflect real-world clinical scenarios. Extensive experiments demonstrate that RET-CLIP outperforms existing benchmarks across eight diverse datasets spanning four critical diagnostic categories: diabetic retinopathy, glaucoma, multiple disease diagnosis, and multi-label classification of multiple diseases, which demonstrate the performance and generality of our foundation model. The sourse code and pre-trained model are available at https://github.com/sStonemason/RET-CLIP.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 25 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

16 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision15 Aug

Other references · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  8. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  9. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  10. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  11. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  12. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
3 more
  1. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  2. 2023ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Datacited by 14
  3. 2023Exploring the Transfer Learning Capabilities of CLIP in Domain Generalization for Diabetic Retinopathycited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 16; 2026: 11.

29 papers.

Sort:

By area:

  1. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  2. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  3. 2024ViLReF: An Expert Knowledge Enabled Vision-Language Retinal Foundation Modelcited by 7
  4. 2025Applications of Large Models in Medicinecited by 6
  5. 2025MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretrainingcited by 5
  6. 2025RetinaLogos: Fine-Grained Synthesis of High-Resolution Retinal Images Through Captionscited by 5
  7. 2025Continual Retinal Vision-Language Pre-training upon Incremental Imaging Modalitiescited by 5
  8. 2025Fine-tuning Vision Language Models with Graph-based Knowledge for Explainable Medical Image Analysiscited by 3
  9. 2025PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learningcited by 3
  10. 2025A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Modelscited by 3
  11. 2025Boosting Medical Visual Understanding From Multi-Granular Language Learningcited by 2
  12. 2025An Explainable Transformer Model for Alzheimer's Disease Detection Using Retinal Imagingcited by 1
17 more
  1. 2025PSScreen: Partially Supervised Multiple Retinal Disease Screeningcited by 1
  2. 2025Native Intelligence Emerges from Large-Scale Clinical Practice: A Retinal Foundation Model with Deployment Efficiencycited by 1
  3. 2026REVEAL: Multimodal Vision-Language Alignment of Retinal Morphometry and Clinical Risks for Incident AD and Dementia Predictioncited by 1
  4. 2026OphIn-500K: Curating Web-Scale Visual Instructions for Scaling Ophthalmic Multimodal Large Language Modelscited by 1
  5. 2025CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignmentcited by 0
  6. 2025UOPSL: Unpaired OCT Predilection Sites Learning for Fundus Image Diagnosis Augmentationcited by 0
  7. 2025Evaluating Fundus-Specific Foundation Models for Diabetic Macular Edema Detectioncited by 0
  8. 2025PSScreen V2: Partially Supervised Multiple Retinal Disease Screeningcited by 0
  9. 2026PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregationcited by 0
  10. 2026Towards Interpretable Foundation Models for Retinal Fundus Imagescited by 0
  11. 2026Bridging Restoration and Diagnosis: A Comprehensive Benchmark for Retinal Fundus Enhancementcited by 0
  12. 2026Scientific Domain Knowledge Improves Vision-Language Fundus Modelscited by 0
  13. 2026Cross-Modal Semantic-Enhanced Diffusion Framework for Diabetic Retinopathy Gradingcited by 0
  14. 2026A Cascaded Edge-Cloud Architecture for Automated Diabetic Retinopathy Screeningcited by 0
  15. 2026REVEAL++: Differentiable Phenotypic Grouping for Vision-Language Retinal Modeling of Alzheimer's Disease Riskcited by 0
  16. 2026Adaptive Conformal Redistribution for Inter-class Transitional Uncertainty in Medical Image Classificationcited by 0
  17. 2026FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CLIP-DR: Textual Knowledge-Guided Diabetic Retinopathy Grading with Ranking-aware Prompting
  2. 2024Enhancing Diagnostic Accuracy in Rare and Common Fundus Diseases with a Knowledge-Rich Vision-Language Model
  3. 2025Interpretable Few-Shot Retinal Disease Diagnosis with Concept-Guided Prompting of Vision-Language Models
  4. 2025RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models
  5. 2026Benchmarking Convolutional, Transformer, Hybrid, and Vision Language Models for Multi Disease Retinal Screening
  6. 2025RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Model
  7. 2025Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening
  8. 2025Beyond CLIP: Knowledge-Enhanced Multimodal Transformers for Cross-Modal Alignment in Diabetic Retinopathy Diagnosis