# Any-to-Any Vision-Language Model for Multimodal X-ray Imaging and Radiological Report Generation

arXiv: https://arxiv.org/abs/2505.01091 · HTML: https://mlsanity.com/paper/2505.01091/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 2 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Any-to-Any Generation via Composable Diffusion](https://mlsanity.com/paper/2305.11846/) (2023)
- [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://mlsanity.com/paper/2211.08332/) (2022)
- [TorchXRayVision: A library of chest X-ray datasets and models](https://mlsanity.com/paper/2111.00595/) (2021)
- [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://mlsanity.com/paper/2004.04092/) (2020)
- [Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains](https://mlsanity.com/paper/2210.04133/) (2022)
- [A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications](https://mlsanity.com/paper/2408.02686/) (2024)
- [Quantifying the Value of Lateral Views in Deep Learning for Chest X-rays](https://mlsanity.com/paper/2002.02582/) (2020)
- [Multimodal Explainability via Latent Shift applied to COVID-19 stratification](https://mlsanity.com/paper/2212.14084/) (2022)
- [Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes](https://mlsanity.com/paper/2204.03772/) (2022)
- [A Deep Learning Approach for Virtual Contrast Enhancement in Contrast Enhanced Spectral Mammography](https://mlsanity.com/paper/2308.00471/) (2023)
- [Multi-Dataset Multi-Task Learning for COVID-19 Prognosis](https://mlsanity.com/paper/2405.13771/) (2024)
- [A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1](https://mlsanity.com/paper/2312.17506/) (2023)

## Built on by (most cited first, 3 of 3)

- [From Alignment to Synthesis: Contrastive Volumetric Grounding for Text-to-CT Generation](https://mlsanity.com/paper/2506.00633/) (2025)
- [Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation](https://mlsanity.com/paper/2603.08305/) (2026)
- [Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching](https://mlsanity.com/paper/2608.08135/) (2026)

## Abstract (arXiv)

Generative models have revolutionized Artificial Intelligence (AI), particularly in multimodal applications. However, adapting these models to the medical domain poses unique challenges due to the complexity of medical data and the stringent need for clinical accuracy. In this work, we introduce a framework specifically designed for multimodal medical data generation. By enabling the generation of multi-view chest X-rays and their associated clinical report, it bridges the gap between general-purpose vision-language models and the specialized requirements of healthcare. Leveraging the MIMIC-CXR dataset, the proposed framework shows superior performance in generating high-fidelity images and semantically coherent reports. Our quantitative evaluation reveals significant results in terms of FID and BLEU scores, showcasing the quality of the generated data. Notably, our framework achieves comparable or even superior performance compared to real data on downstream disease classification tasks, underlining its potential as a tool for medical research and diagnostics. This study highlights the importance of domain-specific adaptations in enhancing the relevance and utility of generative models for clinical applications, paving the way for future advancements in synthetic multimodal medical data generation.