Leveraging Vision-Language Foundation Models to Reveal Hidden Image-Attribute Relationships in Medical Imaging
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Vision-language foundation models (VLMs) have shown impressive performance in guiding image generation through text, with emerging applications in medical imaging. In this work, we are the first to investigate the question: 'Can fine-tuned foundation models help identify critical, and possibly unknown, data properties?' By evaluating our proposed method on a chest x-ray dataset, we show that these models can generate high-resolution, precisely edited images compared to methods that rely on Structural Causal Models (SCMs) according to numerous metrics. For the first time, we demonstrate that fine-tuned VLMs can reveal hidden data relationships that were previously obscured due to available metadata granularity and model capacity limitations. Our experiments demonstrate both the potential of these models to reveal underlying dataset properties while also exposing the limitations of fine-tuned VLMs for accurate image editing and susceptibility to biases and spurious correlations.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 33 references that are in the map. First posted on arXiv 30 Mar 2025.
Builds on
17 of 33 references.
References · 17
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2021Florence: A New Foundation Model for Computer Visioncited by 804
- 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 614
- 2020Deep Structural Causal Models for Tractable Counterfactual Inferencecited by 211
- 2021Explaining in Style: Training a GAN to explain a classifier in StyleSpacecited by 122
- 2021The Causal-Neural Connection: Expressiveness, Learnability, and Inferencecited by 98
- 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 76
- 2023High Fidelity Image Counterfactuals with Probabilistic Causal Modelscited by 53
- 2023ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoderscited by 50
- 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
- 2023Radiology-Llama2: Best-in-Class Large Language Model for Radiologycited by 42
5 more
- 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
- 2020CounteRGAN: Generating Realistic Counterfactuals with Residual Generative Adversarial Netscited by 29
- 2024Counterfactual Image Editingcited by 20
- 2025PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusioncited by 17
- 2023Debiasing Counterfactuals In the Presence of Spurious Correlationscited by 12
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
- 2026Medical Context Distorts Decisions in Clinical Vision Language Models
- 2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains
- 2024Expert-level vision-language foundation model for real-world radiology and comprehensive evaluation
- 2026Cross-modal linkage risk in clinical vision-language models
- 2025Foundation Models in Medical Image Analysis: A Systematic Review and Meta-Analysis
- 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
- 2025Foundation Models in Biomedical Imaging: Turning Hype into Reality