mlsanityData to 14 Oct 2026

Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignment

arXiv 2112.15011 ↗First posted 30 Dec 2021eess.IV cs.CL cs.CVcited by 73

Not on the map. Why?

Abstract (arXiv)

In clinics, a radiology report is crucial for guiding a patient's treatment. However, writing radiology reports is a heavy burden for radiologists. To this end, we present an automatic, multi-modal approach for report generation from a chest x-ray. Our approach, motivated by the observation that the descriptions in radiology reports are highly correlated with specific information of the x-ray images, features two distinct modules: (i) Learned knowledge base: To absorb the knowledge embedded in the radiology reports, we build a knowledge base that can automatically distil and restore medical knowledge from textual embedding without manual labour; (ii) Multi-modal alignment: to promote the semantic alignment among reports, disease labels, and images, we explicitly utilize textual embedding to guide the learning of the visual feature space. We evaluate the performance of the proposed model using metrics from both natural language generation and clinic efficacy on the public IU-Xray and MIMIC-CXR datasets. Our ablation study shows that each module contributes to improving the quality of generated reports. Furthermore, with the assistance of both modules, our approach outperforms state-of-the-art methods over almost all the metrics.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 30 Dec 2021.

Builds on

26 of 41 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  8. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  10. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  11. 2019Attention on Attention for Image Captioningcited by 289
  12. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
14 more
  1. 2020A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promisescited by 231
  2. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  3. 2017On the Automatic Generation of Medical Imaging Reportscited by 216
  4. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 162
  5. 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
  6. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  7. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  8. 2019Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generationcited by 102
  9. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
  10. 2019Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichmentcited by 60
  11. 2016Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotationcited by 59
  12. 2020Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reportscited by 58
  13. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  14. 2020Chest X-ray Report Generation through Fine-Grained Label Learningcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 22; 2025: 27; 2026: 15.

73 papers.

Sort:

By area:

  1. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 101
  2. 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 95
  3. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  4. 2025Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generationcited by 42
  5. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  6. 2024Large-Scale 3D Medical Image Pre-training with Geometric Context Priorscited by 36
  7. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 35
  8. 2022Explainable Deep Learning Methods in Medical Image Classification: A Surveycited by 32
  9. 2024Dia-LLaMA: Towards Large Language Model-driven CT Report Generationcited by 32
  10. 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generationcited by 31
  11. 2023Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generationcited by 26
  12. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiologycited by 25
38 more
  1. 2024Radiology Report Generation via Multi-objective Preference Optimizationcited by 16
  2. 2024Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generationcited by 15
  3. 2024Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patientscited by 14
  4. 2025Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generationcited by 13
  5. 2024Factual Serialization Enhancement: A Key Innovation for Chest X-ray Report Generationcited by 11
  6. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  7. 2024MMSummary: Multimodal Summary Generation for Fetal Ultrasound Videocited by 9
  8. 2025DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generationcited by 9
  9. 2024VM-DDPM: Vision Mamba Diffusion for Medical Image Synthesiscited by 8
  10. 2024Contrastive Learning with Counterfactual Explanations for Radiology Report Generationcited by 8
  11. 2025DDaTR: Dynamic Difference-aware Temporal Residual Network for Longitudinal Radiology Report Generationcited by 8
  12. 2025A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generationcited by 7
  13. 2024Prompt-Guided Generation of Structured Chest X-Ray Report Using a Pre-trained LLMcited by 6
  14. 2024A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Datacited by 6
  15. 2025MRGAgents: A Multi-Agent Framework for Improved Medical Report Generation with Med-LVLMscited by 5
  16. 2025Multimodal Large Language Models for Medical Report Generation via Customized Prompt Tuningcited by 5
  17. 2025A Disease-Aware Dual-Stage Framework for Chest X-ray Report Generationcited by 5
  18. 2023Rethinking Medical Report Generation: Disease Revealing Enhancement with Knowledge Graphcited by 4
  19. 2024EVOKE: Elevating Chest X-ray Report Generation via Multi-View Contrastive Learning and Patient-Specific Knowledgecited by 4
  20. 2024MvKeTR: Chest CT Report Generation with Multi-View Perception and Knowledge Enhancementcited by 4
  21. 2024DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matchingcited by 4
  22. 2025RA-RRG: Multimodal Retrieval-Augmented Radiology Report Generation with Key Phrase Extractioncited by 4
  23. 2025LLM-driven Medical Report Generation via Communication-efficient Heterogeneous Federated Learningcited by 4
  24. 2023Enhanced Knowledge Injection for Radiology Report Generationcited by 3
  25. 2023UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Promptscited by 3
  26. 2024ORID: Organ-Regional Information Driven Framework for Radiology Report Generationcited by 3
  27. 2025MEPNet: Medical Entity-balanced Prompting Network for Brain CT Report Generationcited by 3
  28. 2025Reason Like a Radiologist: Chain-of-Thought and Reinforcement Learning for Verifiable Report Generationcited by 3
  29. 2025Natural Language Generation in Healthcare: A Review of Methods and Applicationscited by 3
  30. 2025Generative AI for Healthcare: Fundamentals, Challenges, and Perspectivescited by 3
  31. 2024MedDiT: A Knowledge-Controlled Diffusion Transformer Framework for Dynamic Medical Image Generation in Virtual Simulated Patientcited by 2
  32. 2025MoColl: Agent-Based Specific and General Model Collaboration for Image Captioningcited by 2
  33. 2025Automated Structured Radiology Report Generation with Rich Clinical Contextcited by 2
  34. 2025Vision-Language Models for Automated 3D PET/CT Report Generationcited by 2
  35. 2024Dynamic Traceback Learning for Medical Report Generationcited by 1
  36. 2024Scene Graph Aided Radiology Report Generationcited by 1
  37. 2025Towards Conditioning Clinical Text Generation for User Controlcited by 1
  38. 2025Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generationcited by 1

50 most cited of 73 All 73 (JSON)