A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Models
Not on the map. Why?
Abstract (arXiv)
Visual impairment represents a major global health challenge, with multimodal imaging providing complementary information that is essential for accurate ophthalmic diagnosis. This comprehensive survey systematically reviews the latest advances in multimodal deep learning methods in ophthalmology up to the year 2025. The review focuses on two main categories: task-specific multimodal approaches and large-scale multimodal foundation models. Task-specific approaches are designed for particular clinical applications such as lesion detection, disease diagnosis, and image synthesis. These methods utilize a variety of imaging modalities including color fundus photography, optical coherence tomography, and angiography. On the other hand, foundation models combine sophisticated vision-language architectures and large language models pretrained on diverse ophthalmic datasets. These models enable robust cross-modal understanding, automated clinical report generation, and decision support. The survey critically examines important datasets, evaluation metrics, and methodological innovations including self-supervised learning, attention-based fusion, and contrastive alignment. It also discusses ongoing challenges such as variability in data, limited annotations, lack of interpretability, and issues with generalizability across different patient populations. Finally, the survey outlines promising future directions that emphasize the use of ultra-widefield imaging and reinforcement learning-based reasoning frameworks to create intelligent, interpretable, and clinically applicable AI systems for ophthalmology.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 49 of its 184 references that are in the map. First posted on arXiv 31 Jul 2025.
Builds on
49 of 184 references.
References · 49
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2023GPT-4 Technical Reportcited by 18,396
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
- 2019REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographscited by 228
- 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 173
- 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
- 2020ROSE: A Retinal OCT-Angiography Vessel Segmentation Dataset and New Modelcited by 78
- 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervisioncited by 78
- 2019Attention Based Glaucoma Detection: A Large-scale Database and CNN Modelcited by 76
- 2021Applications of Deep Learning in Fundus Images: A Reviewcited by 59
37 more
- 2020Self-supervised Feature Learning via Exploiting Multi-modal Data for Retinal Disease Diagnosiscited by 47
- 2022GAMMA Challenge:Glaucoma grAding from Multi-Modality imAgescited by 46
- 2020DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanationcited by 43
- 2018OCTID: Optical Coherence Tomography Image Databasecited by 36
- 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialoguecited by 34
- 2018Towards a glaucoma risk index based on simulated hemodynamics from fundus imagescited by 31
- 2024RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reportscited by 30
- 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
- 2021VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformerscited by 28
- 2022ADAM Challenge: Detecting Age-related Macular Degeneration from Fundus Imagescited by 25
- 2022Cross-modal Clinical Graph Transformer for Ophthalmic Report Generationcited by 24
- 2024MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertisecited by 24
- 2025EyecareGPT: Boosting Comprehensive Ophthalmology Understanding with Tailored Dataset, Benchmark and Modelcited by 22
- 2024VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledgecited by 19
- 2022Uni4Eye: Unified 2D and 3D Self-supervised Pre-training via Masked Image Modeling Transformer for Ophthalmic Image Classificationcited by 18
- 2024EyeFound: A Multimodal Generalist Foundation Model for Ophthalmic Imagingcited by 18
- 2023Harvard Glaucoma Detection and Progression: A Multimodal Multitask Dataset and Generalization-Reinforced Semi-Supervised Learningcited by 17
- 2021Contextualized Keyword Representations for Multi-modal Retinal Image Captioningcited by 16
- 2020Learning Two-Stream CNN for Multi-Modal Age-related Macular Degeneration Categorizationcited by 15
- 2021Multi-Modal Multi-Instance Learning for Retinal Disease Recognitioncited by 14
- 2024TP-DRSeg: Improving Diabetic Retinopathy Lesion Segmentation with Explicit Text-Prompts Assisted SAMcited by 13
- 2019Two-Stream CNN with Loose Pair Training for Multi-modal AMD Categorizationcited by 11
- 2022Multimodal Information Fusion for Glaucoma and DR Classificationcited by 11
- 2020Detecting hidden signs of diabetes in external eye photographscited by 10
- 2024UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modelingcited by 10
- 2025RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Modelscited by 10
- 2019Fully Convolutional Networks for Monocular Retinal Depth Estimation and Optic Disc-Cup Segmentationcited by 7
- 2024ViLReF: An Expert Knowledge Enabled Vision-Language Retinal Foundation Modelcited by 7
- 2025MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysiscited by 7
- 2022Lesion Guided Explainable Few Weak-shot Medical Report Generationcited by 6
- 2023Improved Automatic Diabetic Retinopathy Severity Classification Using Deep Multimodal Fusion of UWF-CFP and OCTA Imagescited by 6
- 2024Non-Invasive to Invasive: Enhancing FFA Synthesis from CFP with a Benchmark Dataset and a Novel Networkcited by 6
- 2024Confidence-aware multi-modality learning for eye disease screeningcited by 5
- 2025MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretrainingcited by 5
- 2020Attention2AngioGAN: Synthesizing Fluorescein Angiography from Retinal Fundus Images using Generative Adversarial Networkscited by 4
- 2024M3T: Multi-Modal Medical Transformer to bridge Clinical Context with Visual Insights for Retinal Image Medical Description Generationcited by 3
- 2025Clinician-Friendly Foundation Models for Ophthalmic Image Diagnostics without Fine-Tuning or Technical Barrierscited by 1
Built on by
Citing papers per year: 2025: 1; 2026: 3.
4 papers.
- 2025Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoningcited by 6
- 2026VOLMO: Versatile and Open Large Models for Ophthalmologycited by 2
- 2026CLEAR-Mamba:Towards Accurate, Adaptive and Trustworthy Multi-Sequence Ophthalmic Angiography Classificationcited by 0
- 2026X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosiscited by 0
- 2026X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis22 Apr 2026
- 2026VOLMO: Versatile and Open Large Models for Ophthalmology25 Mar 2026
- 2026CLEAR-Mamba:Towards Accurate, Adaptive and Trustworthy Multi-Sequence Ophthalmic Angiography Classification28 Jan 2026
- 2025Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning22 Aug 2025
Similar, not cited
Close in wording, no citation either way
- 2025LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology
- 2024LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models
- 2025A Novel Ophthalmic Benchmark for Evaluating Multimodal Large Language Models with Fundus Photographs and OCT Images
- 2024EyeDiff: text-to-image diffusion model improves rare eye disease diagnosis
- 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding
- 2026Can Multimodal Large Language Models Understand OCT?
- 2026FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs
- 2025Benchmarking Next-Generation Reasoning-Focused Large Language Models in Ophthalmology: A Head-to-Head Evaluation on 5,888 Items