# ReXamine-Global: A Framework for Uncovering Inconsistencies in Radiology Report Generation Metrics

arXiv: https://arxiv.org/abs/2408.16208 · HTML: https://mlsanity.com/paper/2408.16208/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 8 of 8)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [MAIRA-1: A specialised large multimodal model for radiology report generation](https://mlsanity.com/paper/2311.13668/) (2023)
- [FineRadScore: A Radiology Report Line-by-Line Evaluation Technique Generating Corrections with Severity Scores](https://mlsanity.com/paper/2405.20613/) (2024)

## Built on by (most cited first, 5 of 5)

- [Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis](https://mlsanity.com/paper/2505.22079/) (2025)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [MedVAL: Toward Expert-Level Medical Text Validation with Language Models](https://mlsanity.com/paper/2507.03152/) (2025)
- [PARROT: An Open Multilingual Radiology Reports Dataset](https://mlsanity.com/paper/2507.22939/) (2025)
- [CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generation](https://mlsanity.com/paper/2601.11488/) (2026)

## Abstract (arXiv)

Given the rapidly expanding capabilities of generative AI models for radiology, there is a need for robust metrics that can accurately measure the quality of AI-generated radiology reports across diverse hospitals. We develop ReXamine-Global, a LLM-powered, multi-site framework that tests metrics across different writing styles and patient populations, exposing gaps in their generalization. First, our method tests whether a metric is undesirably sensitive to reporting style, providing different scores depending on whether AI-generated reports are stylistically similar to ground-truth reports or not. Second, our method measures whether a metric reliably agrees with experts, or whether metric and expert scores of AI-generated report quality diverge for some sites. Using 240 reports from 6 hospitals around the world, we apply ReXamine-Global to 7 established report evaluation metrics and uncover serious gaps in their generalizability. Developers can apply ReXamine-Global when designing new report evaluation metrics, ensuring their robustness across sites. Additionally, our analysis of existing metrics can guide users of those metrics towards evaluation procedures that work reliably at their sites of interest.