# How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study

arXiv: https://arxiv.org/abs/2507.11200 · HTML: https://mlsanity.com/paper/2507.11200/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 4 of its 5 references that are in the map.
- First posted on arXiv 15 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 4 of 4)

- [Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning](https://mlsanity.com/paper/2506.07044/) (2025)
- [HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale](https://mlsanity.com/paper/2406.19280/) (2024)
- [MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning](https://mlsanity.com/paper/2502.19634/) (2025)
- [Disentangling Reasoning and Knowledge in Medical Large Language Models](https://mlsanity.com/paper/2505.11462/) (2025)

## Built on by (most cited first, 3 of 3)

- [Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models](https://mlsanity.com/paper/2604.27720/) (2026)
- [Evaluation of Medical Vision Language Models HuluMed and MedGemma, and general purpose chatbots Gemma 3, ChatGPT Plus, and Claude Pro on real previously unseen wound images](https://mlsanity.com/paper/2606.20723/) (2026)
- [Authority-Preserving Evaluation of Medical Vision-Language Assistants](https://mlsanity.com/paper/2609.22302/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) trained on web-scale corpora excel at natural image tasks and are increasingly repurposed for healthcare; however, their competence in medical tasks remains underexplored. We present a comprehensive evaluation of open-source general-purpose and medically specialised VLMs, ranging from 3B to 72B parameters, across eight benchmarks: MedXpert, OmniMedVQA, PMC-VQA, PathVQA, MMMU, SLAKE, and VQA-RAD. To observe model performance across different aspects, we first separate it into understanding and reasoning components. Three salient findings emerge. First, large general-purpose models already match or surpass medical-specific counterparts on several benchmarks, demonstrating strong zero-shot transfer from natural to medical images. Second, reasoning performance is consistently lower than understanding, highlighting a critical barrier to safe decision support. Third, performance varies widely across benchmarks, reflecting differences in task design, annotation quality, and knowledge demands. No model yet reaches the reliability threshold for clinical deployment, underscoring the need for stronger multimodal alignment and more rigorous, fine-grained evaluation protocols.