# Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning

arXiv: https://arxiv.org/abs/2603.03437 · HTML: https://mlsanity.com/paper/2603.03437/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 3 Mar 2026.

## Builds on (most cited first, 11 of 11)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [m1: Unleash the Potential of Test-Time Scaling for Medical Reasoning with Large Language Models](https://mlsanity.com/paper/2504.00869/) (2025)
- [MedVLThinker: Simple Baselines for Multimodal Medical Reasoning](https://mlsanity.com/paper/2508.02669/) (2025)
- [Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering](https://mlsanity.com/paper/2511.19220/) (2025)

## Built on by (most cited first, 7 of 7)

- [Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs](https://mlsanity.com/paper/2607.03647/) (2026)
- [Do Pathology Vision-Language Models Truly See Pathology?](https://mlsanity.com/paper/2607.21065/) (2026)
- [From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA](https://mlsanity.com/paper/2606.30220/) (2026)
- [Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models](https://mlsanity.com/paper/2604.08815/) (2026)
- [LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering](https://mlsanity.com/paper/2605.09384/) (2026)
- [From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Training](https://mlsanity.com/paper/2609.31450/) (2026)
- [Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists](https://mlsanity.com/paper/2610.04915/) (2026)

## Abstract (arXiv)

Recent work shows that text-only reinforcement learning with verifiable rewards (RLVR) can match or outperform image-text RLVR on multimodal medical VQA benchmarks, suggesting current evaluation protocols may fail to measure causal visual dependence. We introduce a counterfactual evaluation framework using real, blank, and shuffled images across four medical VQA benchmarks: PathVQA, PMC-VQA, SLAKE, and VQA-RAD. Beyond accuracy, we measure Visual Reliance Score (VRS), Image Sensitivity (IS), and introduce Hallucinated Visual Reasoning Rate (HVRR) to detect cases where models generate visual claims despite producing image-invariant answers. Our findings reveal that RLVR improves accuracy while degrading visual grounding: text-only RLVR achieves negative VRS on PathVQA (-0.09), performing better with mismatched images, while image-text RLVR reduces image sensitivity to 39.8% overall despite improving accuracy. On VQA-RAD, both variants achieve 63% accuracy through different mechanisms: text-only RLVR retains 81% performance with blank images, while image-text RLVR shows only 29% image sensitivity. Models generate visual claims in 68-74% of responses, yet 38-43% are ungrounded (HVRR). These findings demonstrate that accuracy-only rewards enable shortcut exploitation, and progress requires grounding-aware evaluation protocols and training objectives that explicitly enforce visual dependence.