# RaTEScore: A Metric for Radiology Report Generation

arXiv: https://arxiv.org/abs/2406.16845 · HTML: https://mlsanity.com/paper/2406.16845/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation](https://mlsanity.com/paper/2306.02022/) (2023)
- [Towards Building Multilingual Language Model for Medicine](https://mlsanity.com/paper/2402.13963/) (2024)
- [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023)
- [Large-Vocabulary Segmentation for Medical Images with Text Prompts](https://mlsanity.com/paper/2312.17183/) (2023)
- [Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python](https://mlsanity.com/paper/2106.07799/) (2021)
- [Large-scale Long-tailed Disease Diagnosis on Radiology Images](https://mlsanity.com/paper/2312.16151/) (2023)
- [LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation](https://mlsanity.com/paper/2404.00998/) (2024)
- [How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain](https://mlsanity.com/paper/2307.00186/) (2023)
- [Knowledge-enhanced Visual-Language Pretraining for Computational Pathology](https://mlsanity.com/paper/2404.09942/) (2024)

## Built on by (most cited first, 30 of 65)

- [Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning](https://mlsanity.com/paper/2506.07044/) (2025)
- [SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers](https://mlsanity.com/paper/2504.00255/) (2025)
- [ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation](https://mlsanity.com/paper/2411.15122/) (2024)
- [AutoRG-Brain: Grounded Report Generation for Brain MRI](https://mlsanity.com/paper/2407.16684/) (2024)
- [Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality](https://mlsanity.com/paper/2609.27607/) (2026)
- [Scaling medical imaging report generation with multimodal reinforcement learning](https://mlsanity.com/paper/2601.17151/) (2026)
- [CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation](https://mlsanity.com/paper/2603.06183/) (2026)
- [Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretation](https://mlsanity.com/paper/2505.21190/) (2025)
- [Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning](https://mlsanity.com/paper/2509.19090/) (2025)
- [RadAgents: Multimodal Agentic Reasoning for Chest X-ray Interpretation with Radiologist-like Workflows](https://mlsanity.com/paper/2509.20490/) (2025)
- [Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphs](https://mlsanity.com/paper/2408.14397/) (2024)
- [Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?](https://mlsanity.com/paper/2410.13523/) (2024)
- [CXRAgent: Director-Orchestrated Multi-Stage Reasoning for Chest X-Ray Interpretation](https://mlsanity.com/paper/2510.21324/) (2025)
- [S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Framework](https://mlsanity.com/paper/2508.02082/) (2025)
- [Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2512.01922/) (2025)
- [XGeM: A Multi-Prompt Foundation Model for Multimodal Medical Data Generation](https://mlsanity.com/paper/2501.04614/) (2025)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of Synthetic Chest Radiographs](https://mlsanity.com/paper/2505.10496/) (2025)
- [Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation](https://mlsanity.com/paper/2512.16201/) (2025)
- [A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation](https://mlsanity.com/paper/2604.00493/) (2026)
- [RadIR: A Scalable Framework for Multi-Grained Medical Image Retrieval via Radiology Report Mining](https://mlsanity.com/paper/2503.04653/) (2025)
- [PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting](https://mlsanity.com/paper/2510.27680/) (2025)
- [ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusion](https://mlsanity.com/paper/2604.09450/) (2026)
- [Exploring the Design Space of 3D MLLMs for CT Report Generation](https://mlsanity.com/paper/2506.21535/) (2025)
- [Clinically Grounded Agent-based Report Evaluation: An Interpretable Metric for Radiology Report Generation](https://mlsanity.com/paper/2508.02808/) (2025)
- [CX-Mind: A Pioneering Multimodal Large Language Model for Interleaved Reasoning in Chest X-ray via Curriculum-Guided Reinforcement Learning](https://mlsanity.com/paper/2508.03733/) (2025)
- [TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis](https://mlsanity.com/paper/2603.05867/) (2026)
- [RadSEM: A Finding-by-Finding Metric for Clinical Consistency in Radiology Reports](https://mlsanity.com/paper/2606.17062/) (2026)
- [Rethinking Whole-Body CT Image Interpretation: An Abnormality-Centric Approach](https://mlsanity.com/paper/2506.03238/) (2025)
- [Recurrent Visual Feature Extraction and Stereo Attentions for CT Report Generation](https://mlsanity.com/paper/2506.19665/) (2025)

## Abstract (arXiv)

This paper introduces a novel, entity-aware metric, termed as Radiological Report (Text) Evaluation (RaTEScore), to assess the quality of medical reports generated by AI models. RaTEScore emphasizes crucial medical entities such as diagnostic outcomes and anatomical details, and is robust against complex medical synonyms and sensitive to negation expressions. Technically, we developed a comprehensive medical NER dataset, RaTE-NER, and trained an NER model specifically for this purpose. This model enables the decomposition of complex radiological reports into constituent medical entities. The metric itself is derived by comparing the similarity of entity embeddings, obtained from a language model, based on their types and relevance to clinical significance. Our evaluations demonstrate that RaTEScore aligns more closely with human preference than existing metrics, validated both on established public benchmarks and our newly proposed RaTE-Eval benchmark.