mlsanityData to 14 Oct 2026

MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models

arXiv 2608.24118 ↗First posted 25 Aug 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language models (VLMs) are increasingly used in clinical pipelines where a chest X-ray is interpreted alongside retrieved reports, preliminary notes, or prior imaging. Existing benchmarks measure whether models answer correctly in isolation, but not whether they preserve a correct image-only decision when plausible context conflicts with the image. We introduce Multi-Context Chest X-ray (MC-CXR), a benchmark of 240 cases expanded into 2,522 instances that isolates context-induced disruption through paired perturbation. Each case fixes the current image and target finding while presenting matched reliable and misleading context across text and prior CXR, with visual overlays where available. MC-CXR defines three task families and two paired metrics, the switch-to-wrong rate and the context-aligned error rate. We evaluate ten VLMs spanning open-source general, medical-domain, and closed-source systems. Image-only accuracy is necessary but insufficient. Mean switch rates range from 45.6-78.1% across misleading textual sources and 35.7-61.7% across misleading visual sources. Among switched predictions, 74.6% align with the misleading label for text versus 17.6% for visual context, a 57.0-point gap (95% CI 50.9-62.8). This text-visual asymmetry is observed under the standardized direct-answer protocol. The dataset is available on PhysioNet.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 41 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

23 of 41 references.

References · 23

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  4. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  7. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  8. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  9. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  10. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 449
  11. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
  12. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
11 more
  1. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  2. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 356
  3. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  4. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
  5. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  6. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 90
  7. 2025Retrieval-Augmented Generation with Conflicting Evidencecited by 63
  8. 2025FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generationcited by 28
  9. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20
  10. 2025Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretationcited by 8
  11. 2026MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rayscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Medical Context Distorts Decisions in Clinical Vision Language Models
  2. 2026Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models
  3. 2024BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays
  4. 2025CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding
  5. 2025XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography
  6. 2026Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays
  7. 2026Vision-language models for chest radiography do not always need the image
  8. 2025CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays