# Knowledge-Augmented Language Models Interpreting Structured Chest X-Ray Findings

arXiv: https://arxiv.org/abs/2505.01711 · HTML: https://mlsanity.com/paper/2505.01711/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 3 May 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024)
- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [Rethinking Visual Dependency in Long-Context Reasoning for Large Vision-Language Models](https://mlsanity.com/paper/2410.19732/) (2024)
- [SCORE: Story Coherence and Retrieval Enhancement for AI Narratives](https://mlsanity.com/paper/2503.23512/) (2025)
- [EventBERT: A Pre-Trained Model for Event Correlation Reasoning](https://mlsanity.com/paper/2110.06533/) (2021)
- [A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions](https://mlsanity.com/paper/2406.03712/) (2024)
- [Multimodal Event Transformer for Image-guided Story Ending Generation](https://mlsanity.com/paper/2301.11357/) (2023)

## Built on by (most cited first, 2 of 2)

- [Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation](https://mlsanity.com/paper/2509.24739/) (2025)
- [Adapting and Evaluating Multimodal Large Language Models for Adolescent Idiopathic Scoliosis Self-Management: A Divide and Conquer Framework](https://mlsanity.com/paper/2509.11645/) (2025)

## Abstract (arXiv)

Automated interpretation of chest X-rays (CXR) is a critical task with the potential to significantly improve clinical workflow and patient care. While recent advances in multimodal foundation models have shown promise, effectively leveraging the full power of large language models (LLMs) for this visual task remains an underexplored area. This paper introduces CXR-TextInter, a novel framework that repurposes powerful text-centric LLMs for CXR interpretation by operating solely on a rich, structured textual representation of the image content, generated by an upstream image analysis pipeline. We augment this LLM-centric approach with an integrated medical knowledge module to enhance clinical reasoning. To facilitate training and evaluation, we developed the MediInstruct-CXR dataset, containing structured image representations paired with diverse, clinically relevant instruction-response examples, and the CXR-ClinEval benchmark for comprehensive assessment across various interpretation tasks. Extensive experiments on CXR-ClinEval demonstrate that CXR-TextInter achieves state-of-the-art quantitative performance across pathology detection, report generation, and visual question answering, surpassing existing multimodal foundation models. Ablation studies confirm the critical contribution of the knowledge integration module. Furthermore, blinded human evaluation by board-certified radiologists shows a significant preference for the clinical quality of outputs generated by CXR-TextInter. Our work validates an alternative paradigm for medical image AI, showcasing the potential of harnessing advanced LLM capabilities when visual information is effectively structured and domain knowledge is integrated.