mlsanityData to 14 Oct 2026

Benchmarking Large Multimodal Models for Ophthalmic Visual Question Answering with OphthalWeChat

arXiv 2505.19624 ↗First posted 26 May 2025cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Purpose: To develop a bilingual multimodal visual question answering (VQA) benchmark for evaluating VLMs in ophthalmology. Methods: Ophthalmic image posts and associated captions published between January 1, 2016, and December 31, 2024, were collected from WeChat Official Accounts. Based on these captions, bilingual question-answer (QA) pairs in Chinese and English were generated using GPT-4o-mini. QA pairs were categorized into six subsets by question type and language: binary (Binary_CN, Binary_EN), single-choice (Single-choice_CN, Single-choice_EN), and open-ended (Open-ended_CN, Open-ended_EN). The benchmark was used to evaluate the performance of three VLMs: GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-72B-Instruct. Results: The final OphthalWeChat dataset included 3,469 images and 30,120 QA pairs across 9 ophthalmic subspecialties, 548 conditions, 29 imaging modalities, and 68 modality combinations. Gemini 2.0 Flash achieved the highest overall accuracy (0.548), outperforming GPT-4o (0.522, P < 0.001) and Qwen2.5-VL-72B-Instruct (0.514, P < 0.001). It also led in both Chinese (0.546) and English subsets (0.550). Subset-specific performance showed Gemini 2.0 Flash excelled in Binary_CN (0.687), Single-choice_CN (0.666), and Single-choice_EN (0.646), while GPT-4o ranked highest in Binary_EN (0.717), Open-ended_CN (BLEU-1: 0.301; BERTScore: 0.382), and Open-ended_EN (BLEU-1: 0.183; BERTScore: 0.240). Conclusions: This study presents the first bilingual VQA benchmark for ophthalmology, distinguished by its real-world context and inclusion of multiple examinations per patient. The dataset reflects authentic clinical decision-making scenarios and enables quantitative evaluation of VLMs, supporting the development of accurate, specialized, and trustworthy AI systems for eye care.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 51 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

15 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025A Novel Ophthalmic Benchmark for Evaluating Multimodal Large Language Models with Fundus Photographs and OCT Images10 Mar
  2. 2025FunBench: Benchmarking Fundus Reading Skills of MLLMs2 Mar
  3. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI6 Aug
  4. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM14 Feb

Other references · 11

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  3. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  4. 2021Medical Visual Question Answering: A Surveycited by 93
  5. 2024Towards Building Multilingual Language Model for Medicinecited by 92
  6. 2021MMBERT: Multimodal BERT Pretraining for Improved Medical VQAcited by 68
  7. 2022Label-efficient Hybrid-supervised Learning for Medical Image Segmentationcited by 10
  8. 2022Consistency-preserving Visual Question Answering in Medical Imagingcited by 10
  9. 2024AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responsescited by 6
  10. 2025DeepSeek-R1 Outperforms Gemini 2.0 Pro, OpenAI o1, and o3-mini in Bilingual Complex Ophthalmology Reasoningcited by 6
  11. 2025Predicting Diabetic Macular Edema Treatment Responses Using OCT: Dataset and Methods of APTOS Competitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026VOLMO: Versatile and Open Large Models for Ophthalmologycited by 2
  2. 2026Towards Clinically Interpretable Ophthalmic VQA via Spatially-Grounded Lesion Evidencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding
  2. 2024Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs
  3. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
  4. 2024WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation
  5. 2026KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examination
  6. 2025Benchmarking Next-Generation Reasoning-Focused Large Language Models in Ophthalmology: A Head-to-Head Evaluation on 5,888 Items
  7. 2025Performance of GPT-5 Frontier Models in Ophthalmology Question Answering
  8. 2026Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence