mlsanityData to 7 Oct 2026

Hearsay: Vision-Language Medical Diagnoses Without an Image

arXiv 2607.26886 ↗First posted 29 Jul 2026cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

When asked to describe a medical image that was never attached, frontier vision-language models do not abstain: they confabulate a diagnosis. We show that this confabulation is not random. It is structured by who the patient is said to be. Across chest X-ray, brain MRI, and dermatology, Claude Opus-4.7, GPT-5.4, and Gemini-3.1-Pro are each queried with only a demographic descriptor and no image, and changing the descriptor systematically shifts the diagnosis returned. Claude concentrates sharply: a 65-year-old white man asking about a skin mole receives Melanoma in nearly every response, and a 32-year-old Black woman asking about her chest X-ray receives a Sarcoidosis diagnosis whose reasoning reads "suspected, based on demographics and classic pattern.'' GPT-5.4's effect is broader, fabricating across every demographic cell we test, most conspicuously naming Sarcoidosis for young Black patients on chest X-ray. Two structural findings sharpen the problem. A hedged regime appears in which the prose acknowledges the missing image while the structured diagnosis field nevertheless names a disease, a dissociation invisible to prose-only audits. And Claude's dermatology effect collapses entirely when 'skin mole' is swapped for 'skin lesion' while GPT-5.4's is preserved, indicating that mirage is a family of distinct failure modes rather than a single phenomenon. Trustworthy VLM deployment in clinical pipelines requires auditing the structured output channel directly, and probe-word sensitivity should be treated as a first-class evaluation dimension

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 4 of its 10 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

4 of 10 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imaging22 Feb

Other references · 3

  1. 2021Reading Race: AI Recognises Patient's Racial Identity In Medical Imagescited by 155
  2. 2026MIRAGE: The Illusion of Visual Understandingcited by 57
  3. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Imagescited by 55

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Vision-language models for chest radiography do not always need the image
  2. 2026Medical Context Distorts Decisions in Clinical Vision Language Models
  3. 2026Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models
  4. 2026Auditing Medical Vision-Language Models on Chest Radiographs: Estimating Reference Agreement Across Institutions
  5. 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations
  6. 2026Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Models
  7. 2026Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Study
  8. 2026How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective