mlsanityData to 14 Oct 2026

RadEval: A framework for radiology text evaluation

arXiv 2509.18030 ↗First posted 22 Sep 2025cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

We introduce RadEval, a unified, open-source framework for evaluating radiology texts. RadEval consolidates a diverse range of metrics, from classic n-gram overlap (BLEU, ROUGE) and contextual measures (BERTScore) to clinical concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT, TemporalEntityF1) and advanced LLM-based evaluators (GREEN). We refine and standardize implementations, extend GREEN to support multiple imaging modalities with a more lightweight model, and pretrain a domain-specific radiology encoder, demonstrating strong zero-shot retrieval performance. We also release a richly annotated expert dataset with over 450 clinically significant error labels and show how different metrics correlate with radiologist judgment. Finally, RadEval provides statistical testing tools and baseline model evaluations across multiple publicly available datasets, facilitating reproducibility and robust benchmarking in radiology report generation.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

23 of 36 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  8. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  9. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  10. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 337
  11. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 319
  12. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
11 more
  1. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 141
  2. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  3. 2024GREEN: Generative Radiology Report Evaluation and Error Notationcited by 128
  4. 2024CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formatscited by 106
  5. 2019Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reportscited by 96
  6. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 83
  7. 2024Libra: Leveraging Temporal Images for Biomedical Radiology Analysiscited by 29
  8. 2025BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLPcited by 28
  9. 2025ReXGradient-160K: A Large-Scale Publicly Available Dataset of Chest Radiographs with Free-text Reportscited by 27
  10. 2024Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"cited by 20
  11. 2024FineRadScore: A Radiology Report Line-by-Line Evaluation Technique Generating Corrections with Severity Scorescited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 18.

22 papers.

Sort:

By area:

  1. 2026Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factualitycited by 11
  2. 2026CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluationcited by 9
  3. 2025CXRAgent: Director-Orchestrated Multi-Stage Reasoning for Chest X-Ray Interpretationcited by 7
  4. 2025RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answeringcited by 6
  5. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  6. 2026Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learningcited by 4
  7. 2025Glaucoma Detection and Structured OCT Report Generation via a Fine-tuned Multimodal Large Language Modelcited by 2
  8. 2026Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomiescited by 2
  9. 2026BTReport: A Framework for Brain Tumor Radiology Report Generation with Clinically Relevant Featurescited by 1
  10. 2026RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matchingcited by 1
  11. 2026MedTextWeaver: Procedural Knowledge Evolution in Agentic Medical Text Editingcited by 0
  12. 2026Radiology Report Generation for Low-Quality X-Ray Imagescited by 0
10 more
  1. 2026CXRMate-2: Structured Multimodal Temporal Embeddings and Tractable Reinforcement Learning for Clinically Acceptable Chest X-ray Radiology Report Generationcited by 0
  2. 2026Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Modelscited by 0
  3. 2026CCS: Clinical Consensus Selection for Radiology Report Generationcited by 0
  4. 2026Learning Directional Semantic Transitions for Longitudinal Chest X-ray Analysiscited by 0
  5. 2026Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reportscited by 0
  6. 2026MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Modelscited by 0
  7. 2026Spatio-Temporal and Clinical Conditioning for Fine-Grained Radiology Report Retrievalcited by 0
  8. 2026NV-Reason-CT: 3D Visual Language Model for CT Analysiscited by 0
  9. 2026QureRadEmbed: Structuring Radiological Similarity through Attribute and Reasoning Supervisioncited by 0
  10. 2026From Sampled Evidence to Radiology Reports: Objectives, Constraints, and Realizationcited by 0