# A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography

arXiv: https://arxiv.org/abs/2502.05926 · HTML: https://mlsanity.com/paper/2502.05926/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024)
- [Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models](https://mlsanity.com/paper/2408.02032/) (2024)
- [Generative AI for Medical Imaging: extending the MONAI Framework](https://mlsanity.com/paper/2307.15208/) (2023)
- [LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation](https://mlsanity.com/paper/2305.11490/) (2023)
- [Multimodal Reasoning with Multimodal Knowledge Graph](https://mlsanity.com/paper/2406.02030/) (2024)
- [Towards Robust Ranker for Text Retrieval](https://mlsanity.com/paper/2206.08063/) (2022)
- [PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generation](https://mlsanity.com/paper/2411.05085/) (2024)
- [Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation](https://mlsanity.com/paper/2312.08078/) (2023)
- [Improving Cross-modal Alignment for Text-Guided Image Inpainting](https://mlsanity.com/paper/2301.11362/) (2023)
- [Style-Aware Contrastive Learning for Multi-Style Image Captioning](https://mlsanity.com/paper/2301.11367/) (2023)
- [EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretation](https://mlsanity.com/paper/2410.09704/) (2024)
- [Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2406.08068/) (2024)
- [Advancing Text-Driven Chest X-Ray Generation with Policy-Based Reinforcement Learning](https://mlsanity.com/paper/2403.06516/) (2024)
- [GAN-based disentanglement learning for chest X-ray rib suppression](https://mlsanity.com/paper/2110.09134/) (2021)
- [Optimising Chest X-Rays for Image Analysis by Identifying and Removing Confounding Factors](https://mlsanity.com/paper/2208.10320/) (2022)
- [CogGPT: Unleashing the Power of Cognitive Dynamics on Large Language Models](https://mlsanity.com/paper/2401.08438/) (2024)
- [Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation](https://mlsanity.com/paper/2410.17918/) (2024)
- [WoLF: Wide-scope Large Language Model Framework for CXR Understanding](https://mlsanity.com/paper/2403.15456/) (2024)
- [An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging](https://mlsanity.com/paper/2406.00667/) (2024)
- [Diff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion model](https://mlsanity.com/paper/2410.20165/) (2024)

## Abstract (arXiv)

The rapid advancements in large language models (LLMs) have unlocked their potential for multimodal tasks, where text and visual data are processed jointly. However, applying LLMs to medical imaging, particularly for chest X-rays (CXR), poses significant challenges due to the need for precise visual-textual alignment and the preservation of critical diagnostic details. In this paper, we propose Multi-Stage Adaptive Vision-Language Tuning (MAViLT), a novel framework designed to enhance multimodal reasoning and generation for CXR understanding. MAViLT incorporates a clinical gradient-weighted tokenization process and a hierarchical fine-tuning strategy, enabling it to generate accurate radiology reports, synthesize realistic CXRs from text, and answer vision-based clinical questions. We evaluate MAViLT on two benchmark datasets, MIMIC-CXR and Indiana University CXR, achieving state-of-the-art results across all tasks. Human evaluations further validate the clinical relevance and utility of MAViLT, making it a robust tool for real-world medical applications. This work demonstrates the feasibility of leveraging LLMs for multimodal medical imaging while addressing key challenges in vision-language integration.