# Knowledge Matters: Radiology Report Generation with General and Specific Knowledge

arXiv: https://arxiv.org/abs/2112.15009 · HTML: https://mlsanity.com/paper/2112.15009/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 30 Dec 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Entity, Relation, and Event Extraction with Contextualized Span Representations](https://mlsanity.com/paper/1909.03546/) (2019)
- [Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning](https://mlsanity.com/paper/1612.01887/) (2016)
- [A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises](https://mlsanity.com/paper/2008.09104/) (2020)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021)
- [When Radiology Report Generation Meets Knowledge Graph](https://mlsanity.com/paper/2002.08277/) (2020)
- [Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation](https://mlsanity.com/paper/1805.08298/) (2018)
- [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018)
- [TorchXRayVision: A library of chest X-ray datasets and models](https://mlsanity.com/paper/2111.00595/) (2021)
- [Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation](https://mlsanity.com/paper/1903.10122/) (2019)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment](https://mlsanity.com/paper/1907.09085/) (2019)
- [Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation](https://mlsanity.com/paper/1603.08486/) (2016)

## Built on by (most cited first, 30 of 68)

- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [GREEN: Generative Radiology Report Evaluation and Error Notation](https://mlsanity.com/paper/2405.03595/) (2024)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation](https://mlsanity.com/paper/2308.12604/) (2023)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [Automated Radiology Report Generation: A Review of Recent Advances](https://mlsanity.com/paper/2405.10842/) (2024)
- [Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation](https://mlsanity.com/paper/2502.20056/) (2025)
- [Large-Scale 3D Medical Image Pre-training with Geometric Context Priors](https://mlsanity.com/paper/2410.09890/) (2024)
- [Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting](https://mlsanity.com/paper/2310.17811/) (2023)
- [Libra: Leveraging Temporal Images for Biomedical Radiology Analysis](https://mlsanity.com/paper/2411.19378/) (2024)
- [Longitudinal Data and a Semantic Similarity Reward for Chest X-Ray Report Generation](https://mlsanity.com/paper/2307.09758/) (2023)
- [Controllable Chest X-Ray Report Generation from Longitudinal Representations](https://mlsanity.com/paper/2310.05881/) (2023)
- [Ultrasound Report Generation with Cross-Modality Feature Alignment via Unsupervised Guidance](https://mlsanity.com/paper/2406.00644/) (2024)
- [Radiology Report Generation via Multi-objective Preference Optimization](https://mlsanity.com/paper/2412.08901/) (2024)
- [Cross-Modal Causal Intervention for Medical Report Generation](https://mlsanity.com/paper/2303.09117/) (2023)
- [Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime](https://mlsanity.com/paper/2303.17644/) (2023)
- [Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generation](https://mlsanity.com/paper/2405.14905/) (2024)
- [Knowledge-enhanced Visual-Language Pretraining for Computational Pathology](https://mlsanity.com/paper/2404.09942/) (2024)
- [Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generation](https://mlsanity.com/paper/2501.03458/) (2025)
- [RadGraph2: Modeling Disease Progression in Radiology Reports via Hierarchical Information Extraction](https://mlsanity.com/paper/2308.05046/) (2023)
- [Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generation](https://mlsanity.com/paper/2404.00588/) (2024)
- [Factual Serialization Enhancement: A Key Innovation for Chest X-ray Report Generation](https://mlsanity.com/paper/2405.09586/) (2024)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [Token Imbalance Adaptation for Radiology Report Generation](https://mlsanity.com/paper/2304.09185/) (2023)
- [Contrastive Learning with Counterfactual Explanations for Radiology Report Generation](https://mlsanity.com/paper/2407.14474/) (2024)
- [DDaTR: Dynamic Difference-aware Temporal Residual Network for Longitudinal Radiology Report Generation](https://mlsanity.com/paper/2505.03401/) (2025)
- [Pre-training on High Definition X-ray Images: An Experimental Study](https://mlsanity.com/paper/2404.17926/) (2024)
- [KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models](https://mlsanity.com/paper/2409.05370/) (2024)
- [A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generation](https://mlsanity.com/paper/2508.09566/) (2025)
- [Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation](https://mlsanity.com/paper/2509.24739/) (2025)

## Abstract (arXiv)

Automatic radiology report generation is critical in clinics which can relieve experienced radiologists from the heavy workload and remind inexperienced radiologists of misdiagnosis or missed diagnose. Existing approaches mainly formulate radiology report generation as an image captioning task and adopt the encoder-decoder framework. However, in the medical domain, such pure data-driven approaches suffer from the following problems: 1) visual and textual bias problem; 2) lack of expert knowledge. In this paper, we propose a knowledge-enhanced radiology report generation approach introduces two types of medical knowledge: 1) General knowledge, which is input independent and provides the broad knowledge for report generation; 2) Specific knowledge, which is input dependent and provides the fine-grained knowledge for report generation. To fully utilize both the general and specific knowledge, we also propose a knowledge-enhanced multi-head attention mechanism. By merging the visual features of the radiology image with general knowledge and specific knowledge, the proposed model can improve the quality of generated reports. Experimental results on two publicly available datasets IU-Xray and MIMIC-CXR show that the proposed knowledge enhanced approach outperforms state-of-the-art image captioning based methods. Ablation studies also demonstrate that both general and specific knowledge can help to improve the performance of radiology report generation.