mlsanityData to 14 Oct 2026

What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

arXiv 2608.25251 ↗First posted 26 Aug 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes. We study this failure mode as a representation-level blind spot relevant to epistemic intelligence, without claiming a formal estimator of epistemic uncertainty. Using NIH ChestXray14 and CheXpert, we first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics. Using PadChest and OpenI, we then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information retained in frozen embeddings. Self-supervised visual initialization improves NIH-to-CheXpert transfer over supervised ImageNet initialization in matched ResNet-18 comparisons, whereas adversarial adaptation is useful only in a narrow regime and becomes unstable as adversarial pressure increases. Multimodal exact-pair retrieval remains low under external OpenI stress testing, and source-proxy information remains recoverable from learned representations. Qualitative nearest-neighbor and Grad-CAM analyses show clinically plausible cross-dataset structure and thoracic attention patterns in many cases, while device-heavy and false-positive cases remain ambiguous. Auxiliary architecture checks are task-dependent and do not support a universal backbone ranking. Overall, the study shows that apparent competence under a single protocol can conceal transfer, alignment, and shortcut-related failure modes, motivating stress-tested evaluation of medical VLMs under distribution shift.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 30 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

22 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training20 Oct

Other references · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  9. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  10. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  11. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  12. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
9 more
  1. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  2. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  3. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 379
  4. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  5. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 309
  6. 2019Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imagingcited by 216
  7. 20203D Self-Supervised Methods for Medical Imagingcited by 134
  8. 2018Deep Learning Predicts Hip Fracture using Confounding Patient and Healthcare Variablescited by 50
  9. 2026Feature-level Site Leakage Reduction for Cross-Hospital Chest X-ray Transfer via Self-Supervised Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
  2. 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?
  3. 2026Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models
  4. 2026Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models
  5. 2026CADRE: Stable, Parameter Efficient Adaptation of Medical Vision Language Models with Bounded Forgetting and Prior Drift
  6. 2026Toward Guarantees for Clinical Reasoning in Vision Language Models via Formal Verification
  7. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  8. 2026CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Models