mlsanityData to 7 Oct 2026

Medical Context Distorts Decisions in Clinical Vision Language Models

arXiv 2605.17436 ↗First posted 17 May 2026cs.CV cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language models (VLMs) are increasingly proposed for clinical decision support, yet their reliability in real-world scenarios that require integrating both visual and textual context from medical records remains poorly characterized. This paper identifies three failure modes: (1) modality over-reliance on text over images, (2) spurious reliance on irrelevant clinical history, and (3) prompt sensitivity across semantically equivalent inputs. We evaluate a diverse set of general-domain and medically-tuned open and closed VLMs on chest x-ray tasks using MIMIC-CXR. By systematically manipulating image-text alignment, clinical history, and prompt formulations, we found that VLM decisions are dominated by the text modality, even when visual evidence is available. Moreover, we observed that VLMs are heavily influenced by irrelevant reports, while minor prompt changes can reverse correct image-based predictions. Our findings underscore the need for explicit safeguards and stress-testing before considering the use of these models in clinical practice.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 37 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

26 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study21 Feb

Other references · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  6. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  7. 2025OpenAI GPT-5 System Cardcited by 910
  8. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  9. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  10. 2025MedGemma Technical Reportcited by 398
  11. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  12. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
13 more
  1. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 97
  2. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 88
  3. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 86
  4. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  5. 2024MedExQA: Medical Question Answering Benchmark with Multiple Explanationscited by 58
  6. 2026MIRAGE: The Illusion of Visual Understandingcited by 57
  7. 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 55
  8. 2022MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Taskscited by 42
  9. 2021Perceptual Score: What Data Modalities Does Your Model Perceive?cited by 35
  10. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 23
  11. 2025Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluationcited by 6
  12. 2024LLMs for clinical risk predictioncited by 3
  13. 2025On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AIcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models
  2. 2026MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models
  3. 2026Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models
  4. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  5. 2026Vision-language models for chest radiography do not always need the image
  6. 2026PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models
  7. 2026Toward Guarantees for Clinical Reasoning in Vision Language Models via Formal Verification
  8. 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations