mlsanityData to 14 Oct 2026

To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models

arXiv 2603.22623 ↗First posted 23 Mar 2026cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language models (VLMs) adapted to the medical domain have shown strong performance on visual question answering benchmarks, yet their robustness against two critical failure modes, hallucination and sycophancy, remains poorly understood, particularly in combination. We evaluate six VLMs (three general-purpose, three medical-specialist) on three medical VQA datasets and uncover a grounding-sycophancy tradeoff: models with the lowest hallucination propensity are the most sycophantic, while the most pressure-resistant model hallucinates more than all medical-specialist models. To characterize this tradeoff, we propose three metrics: L-VASE, a logit-space reformulation of VASE that avoids its double-normalization; CCS, a confidence-calibrated sycophancy score that penalizes high-confidence capitulation; and Clinical Safety Index (CSI), a unified safety index that combines grounding, autonomy, and calibration via a geometric mean. Across 1,151 test cases, no model achieves a CSI above 0.35, indicating that none of the evaluated 7-8B parameter VLMs is simultaneously well-grounded and robust to social pressure. Our findings suggest that joint evaluation of both properties is necessary before these models can be considered for clinical use. Code is available at https://github.com/UTSA-VIRLab/AgreeOrRight

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 24 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

15 of 24 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Benchmarking and Mitigating Sycophancy in Medical Vision Language Models26 Sep
  2. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning26 Feb
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 12

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  5. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  6. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  7. 2024What matters when building vision-language models?cited by 326
  8. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  9. 2025SycEval: Evaluating LLM Sycophancycited by 108
  10. 2024Detecting and Evaluating Medical Hallucinations in Large Vision Language Modelscited by 57
  11. 2025Investigating VLM Hallucination from a Cognitive Psychology Perspective: A First Step Toward Interpretation with Intriguing Observationscited by 4
  12. 2026Tone Matters: The Impact of Linguistic Tone on Hallucination in VLMscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMscited by 1
  2. 2026When Helpfulness Becomes Sycophancy: Sycophancy is a Boundary Failure Between Social Alignment and Epistemic Integrity in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
  2. 2026Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
  3. 2025Are Large Vision Language Models Truly Grounded in Medical Images? Evidence from Italian Clinical Visual Question Answering
  4. 2024SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks
  5. 2025EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models
  6. 2026Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
  7. 2024Beyond the Hype: A dispassionate look at vision-language models in medical scenario
  8. 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding