# Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models

arXiv: https://arxiv.org/abs/2305.03660 · HTML: https://mlsanity.com/paper/2305.03660/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 5 May 2023.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 10 of 10)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation](https://mlsanity.com/paper/2010.10042/) (2020)
- [Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors](https://mlsanity.com/paper/2210.06340/) (2022)

## Built on by (most cited first, 27 of 27)

- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [Exploring the Boundaries of GPT-4 in Radiology](https://mlsanity.com/paper/2310.14573/) (2023)
- [RADAR: Enhancing Radiology Report Generation with Supplementary Knowledge Injection](https://mlsanity.com/paper/2505.14318/) (2025)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models](https://mlsanity.com/paper/2505.11613/) (2025)
- [Towards Interpretable Radiology Report Generation via Concept Bottlenecks using a Multi-Agentic RAG](https://mlsanity.com/paper/2412.16086/) (2024)
- [A Multimodal Multi-Agent Framework for Radiology Report Generation](https://mlsanity.com/paper/2505.09787/) (2025)
- [Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generation](https://mlsanity.com/paper/2406.06496/) (2024)
- [RA-RRG: Multimodal Retrieval-Augmented Radiology Report Generation with Key Phrase Extraction](https://mlsanity.com/paper/2504.07415/) (2025)
- [Evaluation of Language Models in the Medical Context Under Resource-Constrained Settings](https://mlsanity.com/paper/2406.16611/) (2024)
- [LaB-RAG: Label Boosted Retrieval Augmented Generation for Radiology Report Generation](https://mlsanity.com/paper/2411.16523/) (2024)
- [Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports](https://mlsanity.com/paper/2509.21356/) (2025)
- [Designing a Robust Radiology Report Generation System](https://mlsanity.com/paper/2411.01153/) (2024)
- [Towards Automated Situation Awareness: A RAG-Based Framework for Peacebuilding Reports](https://mlsanity.com/paper/2505.10586/) (2025)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Tasks](https://mlsanity.com/paper/2508.12778/) (2025)
- [Analysing Environmental Efficiency in AI for X-Ray Diagnosis](https://mlsanity.com/paper/2511.07436/) (2025)
- [Georeferencing complex relative locality descriptions with large language models](https://mlsanity.com/paper/2512.14228/) (2025)
- [RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2603.10765/) (2026)
- [MedScribe: Clinically Grounded CT Reporting through Agentic Workflows](https://mlsanity.com/paper/2605.01779/) (2026)
- [LLaMA-XR: A Novel Framework for Radiology Report Generation using LLaMA and QLoRA Fine Tuning](https://mlsanity.com/paper/2506.03178/) (2025)
- [Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature](https://mlsanity.com/paper/2511.03261/) (2025)
- [Cross-Modal Clinical Knowledge Integration for Mammography Report Generation](https://mlsanity.com/paper/2605.31093/) (2026)
- [MAC-RRG: Iterative Multi-Agent Collaboration for X-ray Radiology Report Generation](https://mlsanity.com/paper/2609.26124/) (2026)

## Abstract (arXiv)

We propose Retrieval Augmented Generation (RAG) as an approach for automated radiology report writing that leverages multimodally aligned embeddings from a contrastively pretrained vision language model for retrieval of relevant candidate radiology text for an input radiology image and a general domain generative model like OpenAI text-davinci-003, gpt-3.5-turbo and gpt-4 for report generation using the relevant radiology text retrieved. This approach keeps hallucinated generations under check and provides capabilities to generate report content in the format we desire leveraging the instruction following capabilities of these generative models. Our approach achieves better clinical metrics with a BERTScore of 0.2865 (Delta+ 25.88%) and Semb score of 0.4026 (Delta+ 6.31%). Our approach can be broadly relevant for different clinical settings as it allows to augment the automated radiology report generation process with content relevant for that setting while also having the ability to inject user intents and requirements in the prompts as part of the report generation process to modulate the content and format of the generated reports as applicable for that clinical setting.