mlsanityData to 14 Oct 2026

Diagnostic Accuracy of Open-Source Vision-Language Models on Diverse Medical Imaging Tasks

arXiv 2508.01016 ↗First posted 1 Aug 2025eess.IV cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

This retrospective study evaluated five VLMs (Qwen2.5, Phi-4, Gemma3, Llama3.2, and Mistral3.1) using the MedFMC dataset. This dataset includes 22,349 images from 7,461 patients encompassing chest radiography (19 disease multi-label classifications), colon pathology (tumor detection), endoscopy (colorectal lesion identification), neonatal jaundice assessment (skin color-based treatment necessity), and retinal fundoscopy (5-point diabetic retinopathy grading). Diagnostic accuracy was compared in three experimental settings: visual input only, multimodal input, and chain-of-thought reasoning. Model accuracy was assessed against ground truth labels, with statistical comparisons using bootstrapped confidence intervals (p<.05). Qwen2.5 achieved the highest accuracy for chest radiographs (90.4%) and endoscopy images (84.2%), significantly outperforming the other models (p<.001). In colon pathology, Qwen2.5 (69.0%) and Phi-4 (69.6%) performed comparably (p=.41), both significantly exceeding other VLMs (p<.001). Similarly, for neonatal jaundice assessment, Qwen2.5 (58.3%) and Phi-4 (58.1%) showed comparable leading accuracies (p=.93) significantly exceeding their counterparts (p<.001). All models struggled with retinal fundoscopy; Qwen2.5 and Gemma3 achieved the highest, albeit modest, accuracies at 18.6% (comparable, p=.99), significantly better than other tested models (p<.001). Unexpectedly, multimodal input reduced accuracy for some models and modalities, and chain-of-thought reasoning prompts also failed to improve accuracy. The open-source VLMs demonstrated promising diagnostic capabilities, particularly in chest radiograph interpretation. However, performance in complex domains such as retinal fundoscopy was limited, underscoring the need for further development and domain-specific adaptation before widespread clinical application.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 23 references that are in the map. First posted on arXiv 1 Aug 2025.

Builds on

10 of 23 references.

References · 10

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2025Gemma 3 Technical Reportcited by 1,824
  5. 2024Phi-4 Technical Reportcited by 711
  6. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  7. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 173
  8. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  9. 2023MedFMC: A Real-world Dataset and Benchmark For Foundation Model Adaptation in Medical Image Classificationcited by 38
  10. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias
  2. 2026Performance of large language models in the optical diagnosis of colorectal polyps
  3. 2025MedVision: Benchmarking Quantitative Medical Image Analysis
  4. 2026Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?
  5. 2026MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models
  6. 2024Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Models
  7. 2026ViX-Ray: A Vietnamese Chest X-Ray Dataset for Vision-Language Models
  8. 2026Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models