# CXR-LLAVA: a multimodal large language model for interpreting chest X-ray images

arXiv: https://arxiv.org/abs/2310.18341 · HTML: https://mlsanity.com/paper/2310.18341/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 0 of its 22 references that are in the map.
- First posted on arXiv 22 Oct 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Built on by (most cited first, 30 of 37)

- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation](https://mlsanity.com/paper/2405.08807/) (2024)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs](https://mlsanity.com/paper/2311.14656/) (2023)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [Medical MLLM is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Models](https://mlsanity.com/paper/2405.20775/) (2024)
- [Comprehensive Evaluation of Multimodal AI Models in Medical Imaging Diagnosis: From Data Augmentation to Preference-Based Comparison](https://mlsanity.com/paper/2412.05536/) (2024)
- [OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis](https://mlsanity.com/paper/2602.16110/) (2026)
- [Enhancing Human-Computer Interaction in Chest X-ray Analysis using Vision and Language Model with Eye Gaze Patterns](https://mlsanity.com/paper/2404.02370/) (2024)
- [Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume](https://mlsanity.com/paper/2602.24195/) (2026)
- [MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling](https://mlsanity.com/paper/2508.05492/) (2025)
- [Automated Survey Collection with LLM-based Conversational Agents](https://mlsanity.com/paper/2504.02891/) (2025)
- [Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigation](https://mlsanity.com/paper/2604.02543/) (2026)
- [Best Practices for Large Language Models in Radiology](https://mlsanity.com/paper/2412.01233/) (2024)
- [MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation](https://mlsanity.com/paper/2601.09879/) (2026)
- [MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting](https://mlsanity.com/paper/2509.03800/) (2025)
- [MRG-R1: Reinforcement Learning for Clinically Aligned Medical Report Generation](https://mlsanity.com/paper/2512.16145/) (2025)
- [ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusion](https://mlsanity.com/paper/2604.09450/) (2026)
- [Exploring the Design Space of 3D MLLMs for CT Report Generation](https://mlsanity.com/paper/2506.21535/) (2025)
- [A Review of Longitudinal Radiology Report Generation: Dataset Composition, Methods, and Performance Evaluation](https://mlsanity.com/paper/2510.12444/) (2025)
- [Semantics versus Identity: A Divide-and-Conquer Approach towards Adjustable Medical Image De-Identification](https://mlsanity.com/paper/2507.21703/) (2025)
- [Automated Structured Radiology Report Generation with Rich Clinical Context](https://mlsanity.com/paper/2510.00428/) (2025)
- [Opportunities and challenges in the application of large artificial intelligence models in radiology](https://mlsanity.com/paper/2403.16112/) (2024)
- [Explainability Through Human-Centric Design for XAI in Lung Cancer Detection](https://mlsanity.com/paper/2505.09755/) (2025)
- [Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays](https://mlsanity.com/paper/2509.15234/) (2025)
- [Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generation](https://mlsanity.com/paper/2603.16876/) (2026)
- [Visual Instruction-Finetuned Language Model for Versatile Brain MR Image Tasks](https://mlsanity.com/paper/2604.02748/) (2026)
- [LLaMA-XR: A Novel Framework for Radiology Report Generation using LLaMA and QLoRA Fine Tuning](https://mlsanity.com/paper/2506.03178/) (2025)
- [RadEyeVideo: Enhancing general-domain Large Vision Language Model for chest X-ray analysis with video representations of eye gaze](https://mlsanity.com/paper/2507.09097/) (2025)
- [From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature](https://mlsanity.com/paper/2512.02566/) (2025)

## Abstract (arXiv)

Purpose: This study aimed to develop an open-source multimodal large language model (CXR-LLAVA) for interpreting chest X-ray images (CXRs), leveraging recent advances in large language models (LLMs) to potentially replicate the image interpretation skills of human radiologists Materials and Methods: For training, we collected 592,580 publicly available CXRs, of which 374,881 had labels for certain radiographic abnormalities (Dataset 1) and 217,699 provided free-text radiology reports (Dataset 2). After pre-training a vision transformer with Dataset 1, we integrated it with an LLM influenced by the LLAVA network. Then, the model was fine-tuned, primarily using Dataset 2. The model's diagnostic performance for major pathological findings was evaluated, along with the acceptability of radiologic reports by human radiologists, to gauge its potential for autonomous reporting. Results: The model demonstrated impressive performance in test sets, achieving an average F1 score of 0.81 for six major pathological findings in the MIMIC internal test set and 0.62 for seven major pathological findings in the external test set. The model's F1 scores surpassed those of GPT-4-vision and Gemini-Pro-Vision in both test sets. In human radiologist evaluations of the external test set, the model achieved a 72.7% success rate in autonomous reporting, slightly below the 84.0% rate of ground truth reports. Conclusion: This study highlights the significant potential of multimodal LLMs for CXR interpretation, while also acknowledging the performance limitations. Despite these challenges, we believe that making our model open-source will catalyze further research, expanding its effectiveness and applicability in various clinical contexts. CXR-LLAVA is available at https://github.com/ECOFRI/CXR_LLAVA.