# Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation

arXiv: https://arxiv.org/abs/2010.10042 · HTML: https://mlsanity.com/paper/2010.10042/

## Key facts

- Cited by 141 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 30 of 141)

- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [Conformal Language Modeling](https://mlsanity.com/paper/2306.10193/) (2023)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [Exploring scalable medical image encoders beyond text supervision](https://mlsanity.com/paper/2401.10815/) (2024)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications](https://mlsanity.com/paper/2310.18339/) (2023)
- [Advancing Multimodal Medical Capabilities of Gemini](https://mlsanity.com/paper/2405.03162/) (2024)
- [RoentGen: Vision-Language Foundation Model for Chest X-ray Generation](https://mlsanity.com/paper/2211.12737/) (2022)
- [Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards](https://mlsanity.com/paper/2210.12186/) (2022)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [MAIRA-1: A specialised large multimodal model for radiology report generation](https://mlsanity.com/paper/2311.13668/) (2023)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [Cross-modal Prototype Driven Network for Radiology Report Generation](https://mlsanity.com/paper/2207.04818/) (2022)
- [Towards a Visual-Language Foundation Model for Computational Pathology](https://mlsanity.com/paper/2307.12914/) (2023)
- [Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors](https://mlsanity.com/paper/2210.06340/) (2022)
- [MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging](https://mlsanity.com/paper/2410.06542/) (2024)
- [Automated Radiology Report Generation: A Review of Recent Advances](https://mlsanity.com/paper/2405.10842/) (2024)
- [ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning](https://mlsanity.com/paper/2306.06466/) (2023)
- [Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference](https://mlsanity.com/paper/2211.11875/) (2022)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generation](https://mlsanity.com/paper/2303.17579/) (2023)
- [Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts](https://mlsanity.com/paper/2302.08958/) (2023)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset](https://mlsanity.com/paper/2410.00379/) (2024)
- [WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answering](https://mlsanity.com/paper/2407.05603/) (2024)
- [HERGen: Elevating Radiology Report Generation with Longitudinal Data](https://mlsanity.com/paper/2407.15158/) (2024)

## Abstract (arXiv)

Neural image-to-text radiology report generation systems offer the potential to improve radiology reporting by reducing the repetitive process of report drafting and identifying possible medical errors. However, existing report generation systems, despite achieving high performances on natural language generation metrics such as CIDEr or BLEU, still suffer from incomplete and inconsistent generations. Here we introduce two new simple rewards to encourage the generation of factually complete and consistent radiology reports: one that encourages the system to generate radiology domain entities consistent with the reference, and one that uses natural language inference to encourage these entities to be described in inferentially consistent ways. We combine these with the novel use of an existing semantic equivalence metric (BERTScore). We further propose a report generation system that optimizes these rewards via reinforcement learning. On two open radiology report datasets, our system substantially improved the F1 score of a clinical information extraction performance by +22.1 (Delta +63.9%). We further show via a human evaluation and a qualitative analysis that our system leads to generations that are more factually complete and consistent compared to the baselines.