# A Novel Ophthalmic Benchmark for Evaluating Multimodal Large Language Models with Fundus Photographs and OCT Images

arXiv: https://arxiv.org/abs/2503.07094 · HTML: https://mlsanity.com/paper/2503.07094/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 9 of its 51 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 9 of 9)

- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [Mapping global dynamics of benchmark creation and saturation in artificial intelligence](https://mlsanity.com/paper/2203.04592/) (2022)
- [Matching Patients to Clinical Trials with Large Language Models](https://mlsanity.com/paper/2307.15051/) (2023)
- [MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models](https://mlsanity.com/paper/2407.10990/) (2024)
- [A Generic Fundus Image Enhancement Network Boosted by Frequency Self-supervised Representation Learning](https://mlsanity.com/paper/2309.00885/) (2023)
- [Evolving the pulmonary nodules diagnosis from classical approaches to deep learning aided decision support: three decades development course and future prospect](https://mlsanity.com/paper/1901.07858/) (2019)

## Built on by (most cited first, 4 of 4)

- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Benchmarking Large Multimodal Models for Ophthalmic Visual Question Answering with OphthalWeChat](https://mlsanity.com/paper/2505.19624/) (2025)
- [Evaluating Fundus-Specific Foundation Models for Diabetic Macular Edema Detection](https://mlsanity.com/paper/2510.07277/) (2025)
- [Representation learning from OCT images](https://mlsanity.com/paper/2605.02589/) (2026)

## Abstract (arXiv)

In recent years, large language models (LLMs) have demonstrated remarkable potential across various medical applications. Building on this foundation, multimodal large language models (MLLMs) integrate LLMs with visual models to process diverse inputs, including clinical data and medical images. In ophthalmology, LLMs have been explored for analyzing optical coherence tomography (OCT) reports, assisting in disease classification, and even predicting treatment outcomes. However, existing MLLM benchmarks often fail to capture the complexities of real-world clinical practice, particularly in the analysis of OCT images. Many suffer from limitations such as small sample sizes, a lack of diverse OCT datasets, and insufficient expert validation. These shortcomings hinder the accurate assessment of MLLMs' ability to interpret OCT scans and their broader applicability in ophthalmology. Our dataset, curated through rigorous quality control and expert annotation, consists of 439 fundus images and 75 OCT images. Using a standardized API-based framework, we assessed seven mainstream MLLMs and observed significant variability in diagnostic accuracy across different diseases. While some models performed well in diagnosing conditions such as diabetic retinopathy and age-related macular degeneration, they struggled with others, including choroidal neovascularization and myopia, highlighting inconsistencies in performance and the need for further refinement. Our findings emphasize the importance of developing clinically relevant benchmarks to provide a more accurate assessment of MLLMs' capabilities. By refining these models and expanding their scope, we can enhance their potential to transform ophthalmic diagnosis and treatment.