# Designing a Robust Radiology Report Generation System

arXiv: https://arxiv.org/abs/2411.01153 · HTML: https://mlsanity.com/paper/2411.01153/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 62 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation](https://mlsanity.com/paper/1805.08298/) (2018)
- [Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation](https://mlsanity.com/paper/2010.10042/) (2020)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [Clinically Accurate Chest X-Ray Report Generation](https://mlsanity.com/paper/1904.02633/) (2019)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation](https://mlsanity.com/paper/1903.10122/) (2019)
- [Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment](https://mlsanity.com/paper/1907.09085/) (2019)
- [Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation](https://mlsanity.com/paper/1603.08486/) (2016)
- [MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network](https://mlsanity.com/paper/1707.02485/) (2017)
- [Learning to Summarize Radiology Findings](https://mlsanity.com/paper/1809.04698/) (2018)
- [Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generation](https://mlsanity.com/paper/2303.17579/) (2023)
- [Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models](https://mlsanity.com/paper/2305.03660/) (2023)
- [Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rays](https://mlsanity.com/paper/2010.02467/) (2020)
- [Quantifying the Value of Lateral Views in Deep Learning for Chest X-rays](https://mlsanity.com/paper/2002.02582/) (2020)
- [Trust It or Not: Confidence-Guided Automatic Radiology Report Generation](https://mlsanity.com/paper/2106.10887/) (2021)
- [SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models](https://mlsanity.com/paper/2404.17912/) (2024)
- [KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models](https://mlsanity.com/paper/2409.05370/) (2024)

## Built on by (most cited first, 1 of 1)

- [Vision-Language Models for Automated Chest X-ray Interpretation: Leveraging ViT and GPT-2](https://mlsanity.com/paper/2501.12356/) (2025)

## Abstract (arXiv)

Recent advances in deep learning have enabled researchers to explore tasks at the intersection of computer vision and natural language processing, such as image captioning, visual question answering, visual dialogue, and visual language navigation. Taking inspiration from image captioning, the task of radiology report generation aims at automatically generating radiology reports by having a comprehensive understanding of medical images. However, automatically generating radiology reports from medical images is a challenging task due to the complexity, diversity, and nature of medical images. In this paper, we outline the design of a robust radiology report generation system by integrating different modules and highlighting best practices drawing upon lessons from our past work and also from relevant studies in the literature. We also discuss the impact of integrating different components to form a single integrated system. We believe that these best practices, when implemented, could improve automatic radiology report generation, augment radiologists in decision making, and expedite diagnostic workflow, in turn improve healthcare and save human lives.