mlsanityData to 7 Oct 2026

Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions

arXiv 2603.28387 ↗First posted 30 Mar 2026cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Trustworthy clinical AI must use real evidence and avoid relying on surface-level artifacts. We evaluate 12 open-weight vision-language models (VLMs) on two clinical neuroimaging cohorts for binary classification of affective disorders and cognitive decline. Both cohorts include structural magnetic resonance imaging (MRI) acquired under their original research protocols. Prior work does not establish the included neuroimaging inputs as reliable stand-alone diagnostic evidence for the present tasks. Nevertheless, when neuroimaging context is introduced, smaller VLMs gain up to 0.66 F1 under the evaluated augmented conditions, becoming competitive with models an order of magnitude larger. Confidence estimation shows that most of the calibration improvement for the analyzed smaller models occurs after the MRI reference is added to the prompt, before any image is supplied. Our preliminary expert case study finds that faithfulness remains low in every condition examined, with the reviewed model introducing unverified clinical details. Finally, in our single-model intervention, preference alignment suppresses MRI-referencing behavior but reduces the augmented-condition advantage, leaving the underlying issue unresolved. These results caution against reading surface metric gains as evidence of true multimodal integration, with direct implications for clinical VLM deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 47 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

25 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 24

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  9. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  10. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  11. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  12. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
12 more
  1. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  2. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  3. 2022MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Taskscited by 42
  4. 2023How are Prompts Different in Terms of Sensitivity?cited by 34
  5. 2022Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimizationcited by 32
  6. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Modelscited by 31
  7. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactualscited by 27
  8. 2024Open (Clinical) LLMs are Sensitive to Instruction Phrasingscited by 23
  9. 2023Evaluating the Robustness of Discrete Promptscited by 18
  10. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  11. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  12. 2026Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Confident but Unreliable: A Behavioral Safety Audit of Vision-Language Models on Brain MRI
  2. 2026NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders
  3. 2026Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations
  4. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization
  5. 2026Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models
  6. 2025Learning neuroimaging models from health system-scale data
  7. 2026Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs
  8. 2026When Prompts Mislead: Textual Dominance and Diagnostic Bias in MLLMs