# ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment

arXiv: https://arxiv.org/abs/2510.00280 · HTML: https://mlsanity.com/paper/2510.00280/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Builds on (most cited first, 11 of 11)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation](https://mlsanity.com/paper/2401.12208/) (2024)
- [FineRadScore: A Radiology Report Line-by-Line Evaluation Technique Generating Corrections with Severity Scores](https://mlsanity.com/paper/2405.20613/) (2024)
- [RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment](https://mlsanity.com/paper/2501.07525/) (2025)
- [MRScore: Evaluating Radiology Report Generation with LLM-based Reward System](https://mlsanity.com/paper/2404.17778/) (2024)
- [Alice Benchmarks: Connecting Real World Re-Identification with the Synthetic](https://mlsanity.com/paper/2310.04416/) (2023)

## Built on by (most cited first, 8 of 8)

- [Comprehensive language-image pre-training for 3D medical image understanding](https://mlsanity.com/paper/2510.15042/) (2025)
- [CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation](https://mlsanity.com/paper/2603.06183/) (2026)
- [Generating Reports or Repeating Templates? Measuring and Mitigating Template Collapse in 3D CT Report Generation](https://mlsanity.com/paper/2605.30984/) (2026)
- [Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis](https://mlsanity.com/paper/2605.20277/) (2026)
- [Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation](https://mlsanity.com/paper/2608.08713/) (2026)
- [Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning](https://mlsanity.com/paper/2603.08921/) (2026)
- [Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports](https://mlsanity.com/paper/2606.18797/) (2026)
- [Corpus Characterization and Inverse Constitutional Fine-Tuning for Style-Aware Radiology Reports](https://mlsanity.com/paper/2609.14226/) (2026)

## Abstract (arXiv)

Automatically generated radiology reports often receive high scores from existing evaluation metrics but fail to earn clinicians' trust. This gap reveals fundamental flaws in how current metrics assess the quality of generated reports. We rethink the design and evaluation of these metrics and propose a clinically grounded Meta-Evaluation framework. We define clinically grounded criteria spanning clinical alignment and key metric capabilities, including discrimination, robustness, and monotonicity. Using a fine-grained dataset of ground truth and rewritten report pairs annotated with error types, clinical significance labels, and explanations, we systematically evaluate existing metrics and reveal their limitations in interpreting clinical semantics, such as failing to distinguish clinically significant errors, over-penalizing harmless variations, and lacking consistency across error severity levels. Our framework offers guidance for building more clinically reliable evaluation methods.