# A Medical Semantic-Assisted Transformer for Radiographic Report Generation

arXiv: https://arxiv.org/abs/2208.10358 · HTML: https://mlsanity.com/paper/2208.10358/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 22 Aug 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning](https://mlsanity.com/paper/1612.01887/) (2016)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)
- [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022)
- [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021)

## Built on by (most cited first, 30 of 44)

- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens](https://mlsanity.com/paper/2304.02211/) (2023)
- [Automated Radiology Report Generation: A Review of Recent Advances](https://mlsanity.com/paper/2405.10842/) (2024)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis](https://mlsanity.com/paper/2310.20381/) (2023)
- [CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset](https://mlsanity.com/paper/2410.00379/) (2024)
- [Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting](https://mlsanity.com/paper/2310.17811/) (2023)
- [Longitudinal Data and a Semantic Similarity Reward for Chest X-Ray Report Generation](https://mlsanity.com/paper/2307.09758/) (2023)
- [Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reporting](https://mlsanity.com/paper/2307.05766/) (2023)
- [Radiology Report Generation via Multi-objective Preference Optimization](https://mlsanity.com/paper/2412.08901/) (2024)
- [A Systematic Review of Deep Learning-based Research on Radiology Report Generation](https://mlsanity.com/paper/2311.14199/) (2023)
- [Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generation](https://mlsanity.com/paper/2404.00588/) (2024)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [MedRAT: Unpaired Medical Report Generation via Auxiliary Tasks](https://mlsanity.com/paper/2407.03919/) (2024)
- [DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation](https://mlsanity.com/paper/2504.11786/) (2025)
- [KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models](https://mlsanity.com/paper/2409.05370/) (2024)
- [A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Data](https://mlsanity.com/paper/2405.12833/) (2024)
- [MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generation](https://mlsanity.com/paper/2312.02233/) (2023)
- [Multimodal Large Language Models for Medical Report Generation via Customized Prompt Tuning](https://mlsanity.com/paper/2506.15477/) (2025)
- [A Disease-Aware Dual-Stage Framework for Chest X-ray Report Generation](https://mlsanity.com/paper/2511.12259/) (2025)
- [Act Like a Radiologist: Radiology Report Generation across Anatomical Regions](https://mlsanity.com/paper/2305.16685/) (2023)
- [TiBiX: Leveraging Temporal Information for Bidirectional X-ray and Report Generation](https://mlsanity.com/paper/2403.13343/) (2024)
- [Image-aware Evaluation of Generated Medical Reports](https://mlsanity.com/paper/2410.17357/) (2024)
- [Self-Supervised Anatomical Consistency Learning for Vision-Grounded Medical Report Generation](https://mlsanity.com/paper/2509.25963/) (2025)
- [MedCycle: Unpaired Medical Report Generation via Cycle-Consistency](https://mlsanity.com/paper/2403.13444/) (2024)
- [WoLF: Wide-scope Large Language Model Framework for CXR Understanding](https://mlsanity.com/paper/2403.15456/) (2024)
- [Medical Report Generation Is A Multi-label Classification Problem](https://mlsanity.com/paper/2409.00250/) (2024)
- [Online Iterative Self-Alignment for Radiology Report Generation](https://mlsanity.com/paper/2505.11983/) (2025)
- [Automated Structured Radiology Report Generation with Rich Clinical Context](https://mlsanity.com/paper/2510.00428/) (2025)

## Abstract (arXiv)

Automated radiographic report generation is a challenging cross-domain task that aims to automatically generate accurate and semantic-coherence reports to describe medical images. Despite the recent progress in this field, there are still many challenges at least in the following aspects. First, radiographic images are very similar to each other, and thus it is difficult to capture the fine-grained visual differences using CNN as the visual feature extractor like many existing methods. Further, semantic information has been widely applied to boost the performance of generation tasks (e.g. image captioning), but existing methods often fail to provide effective medical semantic features. Toward solving those problems, in this paper, we propose a memory-augmented sparse attention block utilizing bilinear pooling to capture the higher-order interactions between the input fine-grained image features while producing sparse attention. Moreover, we introduce a novel Medical Concepts Generation Network (MCGN) to predict fine-grained semantic concepts and incorporate them into the report generation process as guidance. Our proposed method shows promising performance on the recently released largest benchmark MIMIC-CXR. It outperforms multiple state-of-the-art methods in image captioning and medical report generation.