# Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports

arXiv: https://arxiv.org/abs/2509.21356 · HTML: https://mlsanity.com/paper/2509.21356/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Analyzing and Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2310.00754/) (2023)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [Secrets of RLHF in Large Language Models Part I: PPO](https://mlsanity.com/paper/2307.04964/) (2023)
- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [Improving Chest X-Ray Report Generation by Leveraging Warm Starting](https://mlsanity.com/paper/2201.09405/) (2022)
- [Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors](https://mlsanity.com/paper/2210.06340/) (2022)
- [Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment](https://mlsanity.com/paper/2303.07618/) (2023)
- [Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models](https://mlsanity.com/paper/2305.03660/) (2023)
- [Chest X-ray Report Generation through Fine-Grained Label Learning](https://mlsanity.com/paper/2007.13831/) (2020)
- [RadFlag: A Black-Box Hallucination Detection Method for Medical Vision Language Models](https://mlsanity.com/paper/2411.00299/) (2024)
- [ReXTrust: A Model for Fine-Grained Hallucination Detection in AI-Generated Radiology Reports](https://mlsanity.com/paper/2412.15264/) (2024)
- [Producing radiologist-quality reports for interpretable artificial intelligence](https://mlsanity.com/paper/1806.00340/) (2018)
- [MiDAS: Multi-integrated Domain Adaptive Supervision for Fake News Detection](https://mlsanity.com/paper/2205.09817/) (2022)
- [Fact-Checking of AI-Generated Reports](https://mlsanity.com/paper/2307.14634/) (2023)
- [Extracting and Learning Fine-Grained Labels from Chest Radiographs](https://mlsanity.com/paper/2011.09517/) (2020)
- [Evaluating Automated Radiology Report Quality through Fine-Grained Phrasal Grounding of Clinical Findings](https://mlsanity.com/paper/2412.01031/) (2024)

## Built on by (most cited first, 2 of 2)

- [RadSEM: A Finding-by-Finding Metric for Clinical Consistency in Radiology Reports](https://mlsanity.com/paper/2606.17062/) (2026)
- [Verification Mirage: Mapping the Reliability Boundary of Self-Verification in Medical VQA](https://mlsanity.com/paper/2605.10850/) (2026)

## Abstract (arXiv)

With the emergence of large-scale vision language models (VLM), it is now possible to produce realistic-looking radiology reports for chest X-ray images. However, their clinical translation has been hampered by the factual errors and hallucinations in the produced descriptions during inference. In this paper, we present a novel phrase-grounded fact-checking model (FC model) that detects errors in findings and their indicated locations in automatically generated chest radiology reports. Specifically, we simulate the errors in reports through a large synthetic dataset derived by perturbing findings and their locations in ground truth reports to form real and fake findings-location pairs with images. A new multi-label cross-modal contrastive regression network is then trained on this dataset. We present results demonstrating the robustness of our method in terms of accuracy of finding veracity prediction and localization on multiple X-ray datasets. We also show its effectiveness for error detection in reports of SOTA report generators on multiple datasets achieving a concordance correlation coefficient of 0.997 with ground truth-based verification, thus pointing to its utility during clinical inference in radiology workflows.