# CTest-Metric: A Unified Framework to Assess Clinical Validity of Metrics for CT Report Generation

arXiv: https://arxiv.org/abs/2601.11488 · HTML: https://mlsanity.com/paper/2601.11488/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 16 Jan 2026.

## Builds on (most cited first, 12 of 12)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [GREEN: Generative Radiology Report Evaluation and Error Notation](https://mlsanity.com/paper/2405.03595/) (2024)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [RaTEScore: A Metric for Radiology Report Generation](https://mlsanity.com/paper/2406.16845/) (2024)
- [Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis](https://mlsanity.com/paper/2503.20047/) (2025)
- [Dia-LLaMA: Towards Large Language Model-driven CT Report Generation](https://mlsanity.com/paper/2403.16386/) (2024)
- [Med-2E3: A 2D-Enhanced 3D Medical Multimodal Large Language Model](https://mlsanity.com/paper/2411.12783/) (2024)
- [CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumes](https://mlsanity.com/paper/2408.11965/) (2024)
- [CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generation](https://mlsanity.com/paper/2505.17167/) (2025)
- [ReXamine-Global: A Framework for Uncovering Inconsistencies in Radiology Report Generation Metrics](https://mlsanity.com/paper/2408.16208/) (2024)

## Built on by (most cited first, 2 of 2)

- [RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching](https://mlsanity.com/paper/2609.01470/) (2026)
- [NV-Reason-CT: 3D Visual Language Model for CT Analysis](https://mlsanity.com/paper/2609.27511/) (2026)

## Abstract (arXiv)

In the generative AI era, where even critical medical tasks are increasingly automated, radiology report generation (RRG) continues to rely on suboptimal metrics for quality assessment. Developing domain-specific metrics has therefore been an active area of research, yet it remains challenging due to the lack of a unified, well-defined framework to assess their robustness and applicability in clinical contexts. To address this, we present CTest-Metric, a first unified metric assessment framework with three modules determining the clinical feasibility of metrics for CT RRG. The modules test: (i) Writing Style Generalizability (WSG) via LLM-based rephrasing; (ii) Synthetic Error Injection (SEI) at graded severities; and (iii) Metrics-vs-Expert correlation (MvE) using clinician ratings on 175 "disagreement" cases. Eight widely used metrics (BLEU, ROUGE, METEOR, BERTScore-F1, F1-RadGraph, RaTEScore, GREEN Score, CRG) are studied across seven LLMs built on a CT-CLIP encoder. Using our novel framework, we found that lexical NLG metrics are highly sensitive to stylistic variations; GREEN Score aligns best with expert judgments (Spearman~0.70), while CRG shows negative correlation; and BERTScore-F1 is least sensitive to factual error injection. We will release the framework, code, and allowable portion of the anonymized evaluation data (rephrased/error-injected CT reports), to facilitate reproducible benchmarking and future metric development.