# VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligence

arXiv: https://arxiv.org/abs/2310.04992 · HTML: https://mlsanity.com/paper/2310.04992/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 8 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Embedding Learning via Invariant and Spreading Instance Feature](https://mlsanity.com/paper/1904.03436/) (2019)
- [Predicting Cardiovascular Risk Factors from Retinal Fundus Photographs using Deep Learning](https://mlsanity.com/paper/1708.09843/) (2017)
- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [Learning Two-Stream CNN for Multi-Modal Age-related Macular Degeneration Categorization](https://mlsanity.com/paper/2012.01879/) (2020)
- [From Machine to Machine: An OCT-trained Deep Learning Algorithm for Objective Quantification of Glaucomatous Damage in Fundus Photographs](https://mlsanity.com/paper/1810.10343/) (2018)
- [Predicting optical coherence tomography-derived diabetic macular edema grades from fundus photographs using deep learning](https://mlsanity.com/paper/1810.10342/) (2018)
- [DRAC: Diabetic Retinopathy Analysis Challenge with Ultra-Wide Optical Coherence Tomography Angiography Images](https://mlsanity.com/paper/2304.02389/) (2023)

## Built on by (most cited first, 25 of 25)

- [Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks](https://mlsanity.com/paper/2312.02366/) (2023)
- [MEDCO: Medical Education Copilots Based on A Multi-Agent Framework](https://mlsanity.com/paper/2408.12496/) (2024)
- [Data-Centric Foundation Models in Computational Healthcare: A Survey](https://mlsanity.com/paper/2401.02458/) (2024)
- [Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis](https://mlsanity.com/paper/2312.08592/) (2023)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [Foundation Models in Medical Imaging: A Review and Outlook](https://mlsanity.com/paper/2506.09095/) (2025)
- [MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining](https://mlsanity.com/paper/2501.15798/) (2025)
- [Continual Retinal Vision-Language Pre-training upon Incremental Imaging Modalities](https://mlsanity.com/paper/2506.19320/) (2025)
- [Aria-NeRF: Multimodal Egocentric View Synthesis](https://mlsanity.com/paper/2311.06455/) (2023)
- [GlanceSeg: Real-time microaneurysm lesion segmentation with gaze-map-guided foundation model for early detection of diabetic retinopathy](https://mlsanity.com/paper/2311.08075/) (2023)
- [VIS-MAE: An Efficient Self-supervised Learning Approach on Medical Image Segmentation and Classification](https://mlsanity.com/paper/2402.01034/) (2024)
- [VisionCLIP: An Med-AIGC based Ethical Language-Image Foundation Model for Generalizable Retina Image Analysis](https://mlsanity.com/paper/2403.10823/) (2024)
- [FusionFM: Fusing Eye-specific Foundational Models for Optimized Ophthalmic Diagnosis](https://mlsanity.com/paper/2508.11721/) (2025)
- [Towards Generalist Intelligence in Dentistry: Vision Foundation Models for Oral and Maxillofacial Radiology](https://mlsanity.com/paper/2510.14532/) (2025)
- [Artificial Intelligence for Biomedical Video Generation](https://mlsanity.com/paper/2411.07619/) (2024)
- [Clinician-Friendly Foundation Models for Ophthalmic Image Diagnostics without Fine-Tuning or Technical Barriers](https://mlsanity.com/paper/2504.15928/) (2025)
- [SpineBench: A Clinically Salient, Level-Aware Benchmark Powered by the SpineMed-450k Corpus](https://mlsanity.com/paper/2510.03160/) (2025)
- [Leveraging Multimodal Fusion for Enhanced Diagnosis of Multiple Retinal Diseases in Ultra-wide OCTA](https://mlsanity.com/paper/2311.10331/) (2023)
- [Multimodal, Multi-Disease Medical Imaging Foundation Model (MerMED-FM)](https://mlsanity.com/paper/2507.00185/) (2025)
- [Prompt Mechanisms in Medical Imaging: A Comprehensive Survey](https://mlsanity.com/paper/2507.01055/) (2025)
- [UOPSL: Unpaired OCT Predilection Sites Learning for Fundus Image Diagnosis Augmentation](https://mlsanity.com/paper/2509.08624/) (2025)
- [IRIS: An Intelligent Vision-Language System for Ocular Surface Diseases via Topic Tree and Scene-Driven VQA Generation](https://mlsanity.com/paper/2607.04344/) (2026)
- [Color Fundus Photography Analysis: Co-evolution of Data, Preprocessing, and Modeling toward Multimodal AI](https://mlsanity.com/paper/2607.23972/) (2026)
- [FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs](https://mlsanity.com/paper/2609.33158/) (2026)
- [Med-RADIO: Reducing All Medical Domains Into One via Multi-Teacher Distillation](https://mlsanity.com/paper/2609.37682/) (2026)

## Abstract (arXiv)

We present VisionFM, a foundation model pre-trained with 3.4 million ophthalmic images from 560,457 individuals, covering a broad range of ophthalmic diseases, modalities, imaging devices, and demography. After pre-training, VisionFM provides a foundation to foster multiple ophthalmic artificial intelligence (AI) applications, such as disease screening and diagnosis, disease prognosis, subclassification of disease phenotype, and systemic biomarker and disease prediction, with each application enhanced with expert-level intelligence and accuracy. The generalist intelligence of VisionFM outperformed ophthalmologists with basic and intermediate levels in jointly diagnosing 12 common ophthalmic diseases. Evaluated on a new large-scale ophthalmic disease diagnosis benchmark database, as well as a new large-scale segmentation and detection benchmark database, VisionFM outperformed strong baseline deep neural networks. The ophthalmic image representations learned by VisionFM exhibited noteworthy explainability, and demonstrated strong generalizability to new ophthalmic modalities, disease spectrum, and imaging devices. As a foundation model, VisionFM has a large capacity to learn from diverse ophthalmic imaging data and disparate datasets. To be commensurate with this capacity, in addition to the real data used for pre-training, we also generated and leveraged synthetic ophthalmic imaging data. Experimental results revealed that synthetic data that passed visual Turing tests, can also enhance the representation learning capability of VisionFM, leading to substantial performance gains on downstream ophthalmic AI tasks. Beyond the ophthalmic AI applications developed, validated, and demonstrated in this work, substantial further applications can be achieved in an efficient and cost-effective manner using VisionFM as the foundation.