mlsanityData to 14 Oct 2026

Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering

arXiv 2511.19220 ↗First posted 24 Nov 2025cs.CV cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Large vision language models (VLMs) have achieved impressive performance on medical visual question answering benchmarks, yet their reliance on visual information remains unclear. We investigate whether frontier VLMs demonstrate genuine visual grounding when answering Italian medical questions by testing four state-of-the-art models: Claude Sonnet 4.5, GPT-4o, GPT-5-mini, and Gemini 2.0 flash exp. Using 60 questions from the EuropeMedQA Italian dataset that explicitly require image interpretation, we substitute correct medical images with blank placeholders to test whether models truly integrate visual and textual information. Our results reveal striking variability in visual dependency: GPT-4o shows the strongest visual grounding with a 27.9pp accuracy drop (83.2% [74.6%, 91.7%] to 55.3% [44.1%, 66.6%]), while GPT-5-mini, Gemini, and Claude maintain high accuracy with modest drops of 8.5pp, 2.4pp, and 5.6pp respectively. Analysis of model-generated reasoning reveals confident explanations for fabricated visual interpretations across all models, suggesting varying degrees of reliance on textual shortcuts versus genuine visual analysis. These findings highlight critical differences in model robustness and the need for rigorous evaluation before clinical deployment.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 14 references that are in the map. First posted on arXiv 24 Nov 2025.

Builds on

8 of 14 references.

References · 8

  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  4. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  5. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  6. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  7. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  8. 2025The Illusion of Readiness in Health AIcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoningcited by 7
  2. 2026To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMscited by 3
  3. 2026EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
  2. 2026Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence
  3. 2025Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning
  4. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
  5. 2026Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
  6. 2026To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models
  7. 2024Targeted Visual Prompting for Medical Visual Question Answering
  8. 2024Beyond the Hype: A dispassionate look at vision-language models in medical scenario