mlsanityData to 7 Oct 2026

CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs

arXiv 2605.05810 ↗First posted 7 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

When a chest X-ray shows consolidation but the question asks which finding is present, a medical vision-language model may answer "No consolidation." This is more than an incorrect choice: it is a polarity reversal that emits a clinical statement contradicting the image. We study this failure as negated-option attraction, where a model is drawn to a negated answer option even when it conflicts with both the visual evidence and the question. We introduce CXR-ContraBench (Chest X-Ray Contradiction Benchmark), a diagnostic benchmark spanning internal ReXVQA slices and external OpenI and CheXpert protocols. The benchmark centers on present-finding questions, where selecting "No X" despite visible X creates the main clinical risk, and uses absent-finding questions as secondary tests of whether models copy negated wording. Across CheXpert protocols, the failure is substantial and persistent. On a strict direct presence probe, MedGemma and Qwen2.5-VL reach only 31.49% and 30.21% accuracy, respectively; on a matched 135,754-record CheXpert training-split protocol, both models select negated options on over 62% of presence questions. Chain-of-thought prompting reduces some presence-side reversals but does not eliminate them and can amplify absence-side contradictions. Finally, QCCV-Neg (Question-Conditioned Consistency Verifier for Negation) deterministically repairs the measured polarity-confused subset without retraining, raising MedGemma and Qwen2.5-VL to 96.60% and 95.32% accuracy on the direct presence probe. These results show that standard accuracy can hide a clinically meaningful inference-time polarity failure. Source code and benchmark construction scripts are available at https://github.com/fangzr/cxr-contrabench-code.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 47 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

37 of 47 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering25 May
  2. 2025CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding27 Feb
  3. 2024BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays29 Oct
  4. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI6 Aug
  5. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine6 Aug
  6. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models10 Jun
  7. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models31 Mar
  8. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models14 Feb
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 28

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
16 more
  1. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  2. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  3. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  4. 2025MedGemma Technical Reportcited by 398
  5. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  6. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  7. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  8. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 111
  9. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 108
  10. 2024A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretationcited by 79
  11. 2025Vision-Language Models Do Not Understand Negationcited by 69
  12. 2024ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generationcited by 24
  13. 2025ReXGradient-160K: A Large-Scale Publicly Available Dataset of Chest Radiographs with Free-text Reportscited by 24
  14. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20
  15. 2025Distribution-Aligned Decoding for Efficient LLM Task Adaptationcited by 7
  16. 2025CorBenchX: Large-Scale Chest X-Ray Error Dataset and Vision-Language Model Benchmark for Report Error Correctioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations
  2. 2026MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models
  3. 2025CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays
  4. 2026MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rays
  5. 2026MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence
  6. 2026Reliability Stress Tests and Decision-Time Routing for Chest X-ray Vision-Language Models
  7. 2026Toward Guarantees for Clinical Reasoning in Vision Language Models via Formal Verification
  8. 2026CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays