Utility of Multimodal Large Language Models in Analyzing Chest X-ray with Incomplete Contextual Information
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Background: Large language models (LLMs) are gaining use in clinical settings, but their performance can suffer with incomplete radiology reports. We tested whether multimodal LLMs (using text and images) could improve accuracy and understanding in chest radiography reports, making them more effective for clinical decision support. Purpose: To assess the robustness of LLMs in generating accurate impressions from chest radiography reports using both incomplete data and multimodal data. Material and Methods: We used 300 radiology image-report pairs from the MIMIC-CXR database. Three LLMs (OpenFlamingo, MedFlamingo, IDEFICS) were tested in both text-only and multimodal formats. Impressions were first generated from the full text, then tested by removing 20%, 50%, and 80% of the text. The impact of adding images was evaluated using chest x-rays, and model performance was compared using three metrics with statistical analysis. Results: The text-only models (OpenFlamingo, MedFlamingo, IDEFICS) had similar performance (ROUGE-L: 0.39 vs. 0.21 vs. 0.21; F1RadGraph: 0.34 vs. 0.17 vs. 0.17; F1CheXbert: 0.53 vs. 0.40 vs. 0.40), with OpenFlamingo performing best on complete text (p<0.001). Performance declined with incomplete data across all models. However, adding images significantly boosted the performance of MedFlamingo and IDEFICS (p<0.001), equaling or surpassing OpenFlamingo, even with incomplete text. Conclusion: LLMs may produce low-quality outputs with incomplete radiology data, but multimodal LLMs can improve reliability and support clinical decision-making. Keywords: Large language model; multimodal; semantic analysis; Chest Radiography; Clinical Decision Support;
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 27 references that are in the map. First posted on arXiv 20 Sep 2024.
Builds on
10 of 27 references.
References · 10
- 2023Visual Instruction Tuningcited by 7,869
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
- 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
- 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
- 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 319
- 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
- 2019Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reportscited by 96
- 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 83
- 2021Fairness-Aware PAC Learning from Corrupted Datacited by 18
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Exploring Multimodal Large Language Models for Radiology Report Error-checking
- 2023CXR-LLAVA: a multimodal large language model for interpreting chest X-ray images
- 2023MAIRA-1: A specialised large multimodal model for radiology report generation
- 2024Simplifying Multimodality: Unimodal Approach to Multimodal Challenges in Radiology with General-Domain Large Language Model
- 2026Vision-language models for chest radiography do not always need the image
- 2025Look & Mark: Leveraging Radiologist Eye Fixations and Bounding boxes in Multimodal Large Language Models for Chest X-ray Report Generation
- 2025Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays
- 2024Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancement