mlsanityData to 7 Oct 2026

Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations

arXiv 2608.04885 ↗First posted 5 Aug 2026cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Standard accuracy metrics for VLMs often mask significant reliability failures in sensitive domains. In this work, we utilize a histopathology-validated brain MRI dataset to systematically assess the diagnostic robustness of four VLM families under evidence-preserving perturbations. By reordering anatomical slices and swapping target label positions, we evaluate whether models maintain consistent predictions when clinical evidence remains invariant. Our results reveal significant vulnerabilities in presentation-order stability, with models exhibiting prediction flips in up to 48.9% of cases under simple sequence reversals. We further identify a textual selection bias, where label reordering triggers inconsistent diagnoses in up to 67.8% of cases despite identical visual inputs. Negative-control tests further reveal diagnostic overcommitment: models generate categorical diagnoses in up to 76.1% of cases after expert-annotated lesion slices are removed. These results demonstrate that high accuracy can overestimate clinical reliability, masking sensitivity to sequential presentation and textual framing that is not captured by aggregate accuracy. Our findings highlight the necessity of stability-based metrics for the deployment of VLMs in safety-critical clinical applications. Our evaluation data and code will be made public upon acceptance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 17 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

10 of 17 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations13 Oct
  2. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review4 Mar

Other references · 8

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2025Medical Large Language Model Benchmarks Should Prioritize Construct Validitycited by 36
  3. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  4. 2024MM-R3: On (In-)Consistency of Vision-Language Models (VLMs)cited by 10
  5. 2024Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Modelscited by 5
  6. 2025Questioning the Stability of Visual Question Answeringcited by 4
  7. 2025MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Textcited by 3
  8. 2025On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AIcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Confident but Unreliable: A Behavioral Safety Audit of Vision-Language Models on Brain MRI
  2. 2026Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models
  3. 2026Toward Guarantees for Clinical Reasoning in Vision Language Models via Formal Verification
  4. 2026Rethinking Correctness for Uncertainty Estimation in Clinical Prediction with Vision-Language Models
  5. 2026NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders
  6. 2026MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triage
  7. 2026Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?
  8. 2026Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions