mlsanityData to 14 Oct 2026

Generating Radiology Reports via Memory-driven Transformer

arXiv 2010.16056 ↗First posted 30 Oct 2020cs.CLcited by 368

Not on the map. Why?

Cited by 54 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Medical imaging is frequently used in clinical practice and trials for diagnosis and treatment. Writing imaging reports is time-consuming and can be error-prone for inexperienced radiologists. Therefore, automatically generating radiology reports is highly desired to lighten the workload of radiologists and accordingly promote clinical automation, which is an essential task to apply artificial intelligence to the medical domain. In this paper, we propose to generate radiology reports with memory-driven Transformer, where a relational memory is designed to record key information of the generation process and a memory-driven conditional layer normalization is applied to incorporating the memory into the decoder of Transformer. Experimental results on two prevailing radiology report datasets, IU X-Ray and MIMIC-CXR, show that our proposed approach outperforms previous models with respect to both language generation metrics and clinical evaluations. Particularly, this is the first work reporting the generation results on MIMIC-CXR to the best of our knowledge. Further analyses also demonstrate that our approach is able to generate long reports with necessary medical terms as well as meaningful image-text attention mappings.

Key facts. Cited by 368 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 33; 2023: 74; 2024: 86; 2025: 79; 2026: 81.

368 papers.

Sort:

By area:

  1. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  2. 2023Towards Generalist Biomedical AIcited by 289
  3. 2022Transformers in Medical Imaging: A Surveycited by 245
  4. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
  5. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  6. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  7. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 185
  8. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  9. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 162
  10. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  11. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 140
  12. 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 118
38 more
  1. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  2. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 114
  3. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 106
  4. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 101
  5. 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
  6. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 95
  7. 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 95
  8. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  9. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 83
  10. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 76
  11. 2023MAIRA-1: A specialised large multimodal model for radiology report generationcited by 74
  12. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 73
  13. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
  14. 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
  15. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  16. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  17. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  18. 2023Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Reviewcited by 62
  19. 2023KiUT: Knowledge-injected U-Transformer for Radiology Report Generationcited by 62
  20. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  21. 2023LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generationcited by 57
  22. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  23. 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
  24. 2022Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priorscited by 48
  25. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  26. 2024MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imagingcited by 46
  27. 2022A Medical Semantic-Assisted Transformer for Radiographic Report Generationcited by 44
  28. 2021Progressive Transformer-Based Generation of Radiology Reportscited by 43
  29. 2024HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interactioncited by 43
  30. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  31. 2023ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoningcited by 42
  32. 2025Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generationcited by 42
  33. 2021Weakly Supervised Contrastive Learning for Chest X-Ray Report Generationcited by 41
  34. 2023CLIP in Medical Imaging: A Surveycited by 40
  35. 2023Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generationcited by 37
  36. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  37. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 36
  38. 2025RadGPT: Constructing 3D Image-Text Tumor Datasetscited by 36

50 most cited of 368 All 368 (JSON)