# RadSEM: A Finding-by-Finding Metric for Clinical Consistency in Radiology Reports

arXiv: https://arxiv.org/abs/2606.17062 · HTML: https://mlsanity.com/paper/2606.17062/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [GREEN: Generative Radiology Report Evaluation and Error Notation](https://mlsanity.com/paper/2405.03595/) (2024)
- [RaTEScore: A Metric for Radiology Report Generation](https://mlsanity.com/paper/2406.16845/) (2024)
- [Chest X-ray Report Generation through Fine-Grained Label Learning](https://mlsanity.com/paper/2007.13831/) (2020)
- [MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents](https://mlsanity.com/paper/2511.14439/) (2025)
- [Extracting and Learning Fine-Grained Labels from Chest Radiographs](https://mlsanity.com/paper/2011.09517/) (2020)
- [Evaluating Automated Radiology Report Quality through Fine-Grained Phrasal Grounding of Clinical Findings](https://mlsanity.com/paper/2412.01031/) (2024)
- [Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies](https://mlsanity.com/paper/2503.07306/) (2025)
- [Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reports](https://mlsanity.com/paper/2509.21356/) (2025)

## Built on by (most cited first, 3 of 3)

- [Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality](https://mlsanity.com/paper/2609.27607/) (2026)
- [Measuring the Partial-Credit Gap: A Strict Benchmark on Vietnam's 2025 Convex Marking Scheme](https://mlsanity.com/paper/2608.18336/) (2026)
- [QureRadEmbed: Structuring Radiological Similarity through Attribute and Reasoning Supervision](https://mlsanity.com/paper/2609.33075/) (2026)

## Abstract (arXiv)

Radiology report evaluation must distinguish clinical compatibility from surface similarity, because negation, laterality, or normal-abnormal polarity can reverse a finding. We propose RadSEM (Radiology Sentence-Level Evaluation Metric), a constrained LLM-assisted metric for reference-based evaluation of radiology Findings. RadSEM rewrites reference and generated reports into ordered atomic finding sentences, each expressing one site-finding proposition. It then performs contradiction-constrained many-to-many matching: incompatible pairs such as "effusion" and "no effusion" receive no credit, while compatible granularity differences can receive partial credit. A deterministic stage weights pairs by part-whole and abnormal-detail relationships, counts unmatched findings, and produces an abnormal-focused weighted F1 score. Thus, the LLM supports structured rewriting and local alignment rather than acting as an opaque judge. We evaluate RadSEM with SSREE, a controlled monotonicity stress test built from 2,448 de-identified reports expanded into five graded corruption levels. RadSEM achieves Kendall tau_b of 0.957, all-pairs concordance of 97.8%, adjacent concordance of 95.0%, and strict five-level ordering for 81.9% of reports, outperforming radiology-specific and general text metrics while avoiding the failure in which polarity-inverted reports regain lexical overlap. On the same SSREE set, RadSEM outperforms the Ref-anchored RadSEM-Alt policy, improving adjacent concordance from 90.7% to 95.0% and strict ordering from 67.2% to 81.9%. On a 599-triplet synonym/antonym subset, RadSEM prefers synonyms in 597 cases (99.67%). These results suggest that explicit finding units, contradiction-aware matching, and abnormal-focused deterministic scoring make report scoring more interpretable and sensitive to clinically meaningful errors. Code is available at https://github.com/jdh-algo/RadSEM.