# Image-aware Evaluation of Generated Medical Reports

arXiv: https://arxiv.org/abs/2410.17357 · HTML: https://mlsanity.com/paper/2410.17357/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Builds on (most cited first, 30 of 30)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022)
- [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [When Radiology Report Generation Meets Knowledge Graph](https://mlsanity.com/paper/2002.08277/) (2020)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [Clinically Accurate Chest X-Ray Report Generation](https://mlsanity.com/paper/1904.02633/) (2019)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation](https://mlsanity.com/paper/1903.10122/) (2019)
- [METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens](https://mlsanity.com/paper/2304.02211/) (2023)
- [Joint Learning of Localized Representations from Medical Images and Reports](https://mlsanity.com/paper/2112.02889/) (2021)
- [KiUT: Knowledge-injected U-Transformer for Radiology Report Generation](https://mlsanity.com/paper/2306.11345/) (2023)
- [Cross-modal Prototype Driven Network for Radiology Report Generation](https://mlsanity.com/paper/2207.04818/) (2022)
- [Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation](https://mlsanity.com/paper/2111.04318/) (2021)
- [A Medical Semantic-Assisted Transformer for Radiographic Report Generation](https://mlsanity.com/paper/2208.10358/) (2022)
- [ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning](https://mlsanity.com/paper/2306.06466/) (2023)
- [Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation](https://mlsanity.com/paper/2303.15932/) (2023)
- [Complex Organ Mask Guided Radiology Report Generation](https://mlsanity.com/paper/2311.02329/) (2023)
- [LIMITR: Leveraging Local Information for Medical Image-Text Representation](https://mlsanity.com/paper/2303.11755/) (2023)
- [MedRAT: Unpaired Medical Report Generation via Auxiliary Tasks](https://mlsanity.com/paper/2407.03919/) (2024)
- [MedCycle: Unpaired Medical Report Generation via Cycle-Consistency](https://mlsanity.com/paper/2403.13444/) (2024)

## Built on by (most cited first, 3 of 3)

- [Vision-Language Models for Automated Chest X-ray Interpretation: Leveraging ViT and GPT-2](https://mlsanity.com/paper/2501.12356/) (2025)
- [Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narratives](https://mlsanity.com/paper/2511.09195/) (2025)
- [CAG-VLM: Fine-Tuning of a Large-Scale Model to Recognize Angiographic Images for Next-Generation Diagnostic Systems](https://mlsanity.com/paper/2505.04964/) (2025)

## Abstract (arXiv)

The paper proposes a novel evaluation metric for automatic medical report generation from X-ray images, VLScore. It aims to overcome the limitations of existing evaluation methods, which either focus solely on textual similarities, ignoring clinical aspects, or concentrate only on a single clinical aspect, the pathology, neglecting all other factors. The key idea of our metric is to measure the similarity between radiology reports while considering the corresponding image. We demonstrate the benefit of our metric through evaluation on a dataset where radiologists marked errors in pairs of reports, showing notable alignment with radiologists' judgments. In addition, we provide a new dataset for evaluating metrics. This dataset includes well-designed perturbations that distinguish between significant modifications (e.g., removal of a diagnosis) and insignificant ones. It highlights the weaknesses in current evaluation metrics and provides a clear framework for analysis.