mlsanityData to 7 Oct 2026

Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning

arXiv 2603.03437 ↗First posted 3 Mar 2026cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent work shows that text-only reinforcement learning with verifiable rewards (RLVR) can match or outperform image-text RLVR on multimodal medical VQA benchmarks, suggesting current evaluation protocols may fail to measure causal visual dependence. We introduce a counterfactual evaluation framework using real, blank, and shuffled images across four medical VQA benchmarks: PathVQA, PMC-VQA, SLAKE, and VQA-RAD. Beyond accuracy, we measure Visual Reliance Score (VRS), Image Sensitivity (IS), and introduce Hallucinated Visual Reasoning Rate (HVRR) to detect cases where models generate visual claims despite producing image-invariant answers. Our findings reveal that RLVR improves accuracy while degrading visual grounding: text-only RLVR achieves negative VRS on PathVQA (-0.09), performing better with mismatched images, while image-text RLVR reduces image sensitivity to 39.8% overall despite improving accuracy. On VQA-RAD, both variants achieve 63% accuracy through different mechanisms: text-only RLVR retains 81% performance with blank images, while image-text RLVR shows only 29% image sensitivity. Models generate visual claims in 68-74% of responses, yet 38-43% are ungrounded (HVRR). These findings demonstrate that accuracy-only rewards enable shortcut exploitation, and progress requires grounding-aware evaluation protocols and training objectives that explicitly enforce visual dependence.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 11 of its 14 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

11 of 14 references.

References · 11

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  4. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  5. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  6. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  7. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  8. 2017Deep Reinforcement Learning: An Overviewcited by 327
  9. 2025m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning with Large Language Modelscited by 42
  10. 2025MedVLThinker: Simple Baselines for Multimodal Medical Reasoningcited by 26
  11. 2025Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answeringcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMscited by 6
  2. 2026Do Pathology Vision-Language Models Truly See Pathology?cited by 2
  3. 2026From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQAcited by 1
  4. 2026Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Modelscited by 0
  5. 2026LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answeringcited by 0
  6. 2026From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning
  2. 2026When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains
  3. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning
  4. 2026Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning
  5. 2026Visual sensitivity is not claim retractability: persistence-aware credit assignment for multimodal reinforcement learning
  6. 2026MJ1: Multimodal Judgment via Grounded Verification
  7. 2026Faithful-MR1: Faithful Multimodal Reasoning via Anchoring and Reinforcing Visual Attention
  8. 2026Med-R2: An Adversarial Benchmark for Evidence-Grounded Reasoning in Medical VLMs