# M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation

arXiv: https://arxiv.org/abs/2408.16213 · HTML: https://mlsanity.com/paper/2408.16213/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 49 of its 67 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 49)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Can AI help in screening Viral and COVID-19 pneumonia?](https://mlsanity.com/paper/2003.13145/) (2020)
- [Honeybee: Locality-enhanced Projector for Multimodal LLM](https://mlsanity.com/paper/2312.06742/) (2023)
- [When Radiology Report Generation Meets Knowledge Graph](https://mlsanity.com/paper/2002.08277/) (2020)
- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [Exploring scalable medical image encoders beyond text supervision](https://mlsanity.com/paper/2401.10815/) (2024)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens](https://mlsanity.com/paper/2304.02211/) (2023)

## Built on by (most cited first, 17 of 17)

- [MedRAX: Medical Reasoning Agent for Chest X-ray](https://mlsanity.com/paper/2502.02673/) (2025)
- [Libra: Leveraging Temporal Images for Biomedical Radiology Analysis](https://mlsanity.com/paper/2411.19378/) (2024)
- [RadVLM: A Multitask Conversational Vision-Language Model for Radiology](https://mlsanity.com/paper/2502.03333/) (2025)
- [Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis](https://mlsanity.com/paper/2509.09254/) (2025)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [RADAR: Enhancing Radiology Report Generation with Supplementary Knowledge Injection](https://mlsanity.com/paper/2505.14318/) (2025)
- [A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generation](https://mlsanity.com/paper/2508.09566/) (2025)
- [RA-RRG: Multimodal Retrieval-Augmented Radiology Report Generation with Key Phrase Extraction](https://mlsanity.com/paper/2504.07415/) (2025)
- [A Review of Longitudinal Radiology Report Generation: Dataset Composition, Methods, and Performance Evaluation](https://mlsanity.com/paper/2510.12444/) (2025)
- [MM-NeuroOnco: A Multimodal Benchmark and Instruction Dataset for MRI-Based Brain Tumor Diagnosis](https://mlsanity.com/paper/2602.22955/) (2026)
- [Recurrent Visual Feature Extraction and Stereo Attentions for CT Report Generation](https://mlsanity.com/paper/2506.19665/) (2025)
- [A DeepSeek-Powered AI System for Automated Chest Radiograph Interpretation in Clinical Practice](https://mlsanity.com/paper/2512.20344/) (2025)
- [A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering](https://mlsanity.com/paper/2508.02841/) (2025)
- [Mind the Tool Failures: Achieving Synergistic Tool Gains for Medical Agents](https://mlsanity.com/paper/2605.26691/) (2026)
- [CR3G: Causal Reasoning for Patient-Centric Explanations in Radiology Report Generation](https://mlsanity.com/paper/2512.11830/) (2025)
- [CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretation](https://mlsanity.com/paper/2608.30758/) (2026)
- [X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning](https://mlsanity.com/paper/2610.16811/) (2026)

## Abstract (arXiv)

The rapid evolution of artificial intelligence, especially in large language models (LLMs), has significantly impacted various domains, including healthcare. In chest X-ray (CXR) analysis, previous studies have employed LLMs, but with limitations: either underutilizing the multi-tasking capabilities of LLMs or lacking clinical accuracy. This paper presents M4CXR, a multi-modal LLM designed to enhance CXR interpretation. The model is trained on a visual instruction-following dataset that integrates various task-specific datasets in a conversational format. As a result, the model supports multiple tasks such as medical report generation (MRG), visual grounding, and visual question answering (VQA). M4CXR achieves state-of-the-art clinical accuracy in MRG by employing a chain-of-thought prompting strategy, in which it identifies findings in CXR images and subsequently generates corresponding reports. The model is adaptable to various MRG scenarios depending on the available inputs, such as single-image, multi-image, and multi-study contexts. In addition to MRG, M4CXR performs visual grounding at a level comparable to specialized models and also demonstrates outstanding performance in VQA. Both quantitative and qualitative assessments reveal M4CXR's versatility in MRG, visual grounding, and VQA, while consistently maintaining clinical accuracy.