# GREEN: Generative Radiology Report Evaluation and Error Notation

arXiv: https://arxiv.org/abs/2405.03595 · HTML: https://mlsanity.com/paper/2405.03595/

## Key facts

- Cited by 128 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 6 May 2024.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards](https://mlsanity.com/paper/2210.12186/) (2022)
- [BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients](https://mlsanity.com/paper/2006.01174/) (2020)
- [A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation](https://mlsanity.com/paper/2401.12208/) (2024)
- [Knowledge Matters: Radiology Report Generation with General and Specific Knowledge](https://mlsanity.com/paper/2112.15009/) (2021)
- [Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors](https://mlsanity.com/paper/2210.06340/) (2022)
- [Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generation](https://mlsanity.com/paper/2303.17579/) (2023)
- [Automated Generation of Accurate & Fluent Medical X-ray Reports](https://mlsanity.com/paper/2108.12126/) (2021)
- [Improving Broad-Coverage Medical Entity Linking with Semantic Type Prediction and Large-Scale Datasets](https://mlsanity.com/paper/2005.00460/) (2020)

## Built on by (most cited first, 30 of 128)

- [Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding](https://mlsanity.com/paper/2501.14548/) (2025)
- [VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge](https://mlsanity.com/paper/2411.12915/) (2024)
- [SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers](https://mlsanity.com/paper/2504.00255/) (2025)
- [Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation](https://mlsanity.com/paper/2502.20056/) (2025)
- [ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation](https://mlsanity.com/paper/2411.15122/) (2024)
- [RadVLM: A Multitask Conversational Vision-Language Model for Radiology](https://mlsanity.com/paper/2502.03333/) (2025)
- [RadEval: A framework for radiology text evaluation](https://mlsanity.com/paper/2509.18030/) (2025)
- [Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"](https://mlsanity.com/paper/2409.16603/) (2024)
- [Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-training](https://mlsanity.com/paper/2508.03742/) (2025)
- [Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generation](https://mlsanity.com/paper/2501.03458/) (2025)
- [R2GenCSR: Mining Contextual and Residual Information for LLMs-based Radiology Report Generation](https://mlsanity.com/paper/2408.09743/) (2024)
- [RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment](https://mlsanity.com/paper/2501.07525/) (2025)
- [FMBench: Benchmarking Fairness in Multimodal Large Language Models on Medical Tasks](https://mlsanity.com/paper/2410.01089/) (2024)
- [Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality](https://mlsanity.com/paper/2609.27607/) (2026)
- [Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation](https://mlsanity.com/paper/2406.07146/) (2024)
- [PriorRG: Prior-Guided Contrastive Pre-training and Coarse-to-Fine Decoding for Chest X-ray Report Generation](https://mlsanity.com/paper/2508.05353/) (2025)
- [CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation](https://mlsanity.com/paper/2603.06183/) (2026)
- [The Impact of Auxiliary Patient Data on Automated Chest X-Ray Report Generation and How to Incorporate It](https://mlsanity.com/paper/2406.13181/) (2024)
- [DDaTR: Dynamic Difference-aware Temporal Residual Network for Longitudinal Radiology Report Generation](https://mlsanity.com/paper/2505.03401/) (2025)
- [Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretation](https://mlsanity.com/paper/2505.21190/) (2025)
- [RadAgents: Multimodal Agentic Reasoning for Chest X-ray Interpretation with Radiologist-like Workflows](https://mlsanity.com/paper/2509.20490/) (2025)
- [Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphs](https://mlsanity.com/paper/2408.14397/) (2024)
- [CXRAgent: Director-Orchestrated Multi-Stage Reasoning for Chest X-Ray Interpretation](https://mlsanity.com/paper/2510.21324/) (2025)
- [A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Data](https://mlsanity.com/paper/2405.12833/) (2024)
- [CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generation](https://mlsanity.com/paper/2505.17167/) (2025)
- [Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learning](https://mlsanity.com/paper/2505.17436/) (2025)
- [Region-Aware Multimodal Large Language Model via SlowFast Tokenization and Pseudo-Mask Guidance for 3D CT Report Generation](https://mlsanity.com/paper/2506.23102/) (2025)
- [S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Framework](https://mlsanity.com/paper/2508.02082/) (2025)
- [RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography](https://mlsanity.com/paper/2604.15231/) (2026)
- [Best Practices for Large Language Models in Radiology](https://mlsanity.com/paper/2412.01233/) (2024)

## Abstract (arXiv)

Evaluating radiology reports is a challenging problem as factual correctness is extremely important due to the need for accurate medical communication about medical images. Existing automatic evaluation metrics either suffer from failing to consider factual correctness (e.g., BLEU and ROUGE) or are limited in their interpretability (e.g., F1CheXpert and F1RadGraph). In this paper, we introduce GREEN (Generative Radiology Report Evaluation and Error Notation), a radiology report generation metric that leverages the natural language understanding of language models to identify and explain clinically significant errors in candidate reports, both quantitatively and qualitatively. Compared to current metrics, GREEN offers: 1) a score aligned with expert preferences, 2) human interpretable explanations of clinically significant errors, enabling feedback loops with end-users, and 3) a lightweight open-source method that reaches the performance of commercial counterparts. We validate our GREEN metric by comparing it to GPT-4, as well as to error counts of 6 experts and preferences of 2 experts. Our method demonstrates not only higher correlation with expert error counts, but simultaneously higher alignment with expert preferences when compared to previous approaches.