mlsanityData to 7 Oct 2026

VisionUnite: A Vision-Language Foundation Model for Ophthalmology Enhanced with Clinical Knowledge

arXiv 2408.02865 ↗First posted 5 Aug 2024eess.IV cs.AI cs.CLcited by 19

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

The need for improved diagnostic methods in ophthalmology is acute, especially in the underdeveloped regions with limited access to specialists and advanced equipment. Therefore, we introduce VisionUnite, a novel vision-language foundation model for ophthalmology enhanced with clinical knowledge. VisionUnite has been pretrained on an extensive dataset comprising 1.24 million image-text pairs, and further refined using our proposed MMFundus dataset, which includes 296,379 high-quality fundus image-text pairs and 889,137 simulated doctor-patient dialogue instances. Our experiments indicate that VisionUnite outperforms existing generative foundation models such as GPT-4V and Gemini Pro. It also demonstrates diagnostic capabilities comparable to junior ophthalmologists. VisionUnite performs well in various clinical scenarios including open-ended multi-disease diagnosis, clinical explanation, and patient interaction, making it a highly versatile tool for initial ophthalmic disease screening. VisionUnite can also serve as an educational aid for junior ophthalmologists, accelerating their acquisition of knowledge regarding both common and underrepresented ophthalmic conditions. VisionUnite represents a significant advancement in ophthalmology, with broad implications for diagnostics, medical education, and understanding of disease mechanisms. The source code is at https://github.com/HUANGLIZI/VisionUnite.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 98 references that are in the map. First posted on arXiv 5 Aug 2024.

Builds on

27 of 98 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 26

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2016Language Modeling with Gated Convolutional Networkscited by 688
14 more
  1. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  2. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  3. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  4. 2019REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographscited by 226
  5. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  6. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  7. 2019Attention Based Glaucoma Detection: A Large-scale Database and CNN Modelcited by 75
  8. 2022GAMMA Challenge:Glaucoma grAding from Multi-Modality imAgescited by 45
  9. 2020DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanationcited by 43
  10. 2023AIROGS: Artificial Intelligence for RObust Glaucoma Screening Challengecited by 35
  11. 2020G1020: A Benchmark Retinal Fundus Image Dataset for Computer-Aided Glaucoma Detectioncited by 31
  12. 2022ADAM Challenge: Detecting Age-related Macular Degeneration from Fundus Imagescited by 25
  13. 2022Multi-Label Retinal Disease Classification using Transformerscited by 18
  14. 2023Benchmarking Deep Learning Frameworks for Automated Diagnosis of Ocular Toxoplasmosis: A Comprehensive Approach to Classification and Segmentationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 8.

19 papers.

Sort:

By area:

  1. 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflowcited by 71
  2. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  3. 2024Mask of truth: model sensitivity to unexpected regions of medical imagescited by 6
  4. 2025Constructing Ophthalmic MLLM for Positioning-diagnosis Collaboration Through Clinical Cognitive Chain Reasoningcited by 6
  5. 2025MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretrainingcited by 5
  6. 2025Continual Retinal Vision-Language Pre-training upon Incremental Imaging Modalitiescited by 5
  7. 2025A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Modelscited by 3
  8. 2026DermoGPT: Open Weights and Open Data for Morphology-Grounded Dermatological Reasoning MLLMscited by 3
  9. 2025Boosting Medical Visual Understanding From Multi-Granular Language Learningcited by 2
  10. 2025RetSTA: An LLM-Based Approach for Standardizing Clinical Fundus Image Reportscited by 1
  11. 2026Scale-aware Adaptive Supervised Network with Limited Medical Annotationscited by 1
  12. 2026ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-trainingcited by 1
7 more
  1. 2025UOPSL: Unpaired OCT Predilection Sites Learning for Fundus Image Diagnosis Augmentationcited by 0
  2. 2025RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Modelcited by 0
  3. 2026Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysiscited by 0
  4. 2026Logi-PAR: Logic-Infused Patient Activity Recognition via Differentiable Rulecited by 0
  5. 2026ConceptSMILE: Auditing the Trustworthiness of Concept-Based Explainable AIcited by 0
  6. 2026An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photographycited by 0
  7. 2026MedRSI: Recursive Self-Improvement for Medical Agents via Clinically Aligned Self-Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VOLMO: Versatile and Open Large Models for Ophthalmology
  2. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue
  3. 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding
  4. 2024Specialized curricula for training vision-language models in retinal image analysis
  5. 2026Scientific Domain Knowledge Improves Vision-Language Fundus Models
  6. 2024LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models
  7. 2025LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology
  8. 2025EyecareGPT: Boosting Comprehensive Ophthalmology Understanding with Tailored Dataset, Benchmark and Model