# PathReportEval: A Systematic Benchmark for Pathology Report Generation

arXiv: https://arxiv.org/abs/2607.18448 · HTML: https://mlsanity.com/paper/2607.18448/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 20 Jul 2026.

## Builds on (most cited first, 13 of 13)

- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Multimodal Whole Slide Foundation Model for Pathology](https://mlsanity.com/paper/2411.19666/) (2024)
- [Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation](https://mlsanity.com/paper/2010.10042/) (2020)
- [Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images](https://mlsanity.com/paper/2306.07831/) (2023)
- [Accelerating Data Processing and Benchmarking of AI Models for Pathology](https://mlsanity.com/paper/2502.06750/) (2025)
- [Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles](https://mlsanity.com/paper/2103.05121/) (2021)
- [HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction](https://mlsanity.com/paper/2403.05396/) (2024)
- [WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images](https://mlsanity.com/paper/2311.16480/) (2023)
- [CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation](https://mlsanity.com/paper/2603.06183/) (2026)
- [Historical Report Guided Bi-modal Concurrent Learning for Pathology Report Generation](https://mlsanity.com/paper/2506.18658/) (2025)
- [Pathology Report Generation and Multimodal Representation Learning for Cutaneous Melanocytic Lesions](https://mlsanity.com/paper/2502.19293/) (2025)
- [S2D-ALIGN: Shallow-to-Deep Auxiliary Learning for Anatomically-Grounded Radiology Report Generation](https://mlsanity.com/paper/2511.11066/) (2025)
- [Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation](https://mlsanity.com/paper/2605.01144/) (2026)

## Abstract (arXiv)

Pathology report generation from whole-slide images (WSIs) is a rapidly growing multimodal learning problem, yet progress is difficult to measure because existing studies use heterogeneous datasets, model settings, visual encoders, and evaluation protocols. Moreover, commonly used natural language generation metrics, including BLEU, ROUGE, and METEOR, primarily reward lexical similarity and often fail to detect clinically consequential errors such as omitted diagnoses, hallucinated findings, or discordant tumor attributes. We present a standardized benchmark and evaluation framework for pathology report generation. The benchmark evaluates four representative methods across three datasets (TCGA, HistAI, and REG 2025) using three pathology foundation encoders (CONCHv1.5, UNI2-h, and H-Optimus-1). Our framework standardizes preprocessing, feature extraction, training, decoding, and evaluation, enabling fair comparison across models while providing a modular platform for integrating new methods, datasets, and encoders. A central contribution is the Clinical Report Quality Score (CRQS), a clinically grounded metric for evaluating factual correctness. CRQS maps reference and generated reports into structured clinical attributes and measures four complementary dimensions: clinical fact coverage, key information recall, hallucination rate, and clinical discordance, producing both an overall score and interpretable sub-scores. Experiments demonstrate that conventional language-generation metrics are weakly aligned with clinical correctness and frequently overestimate report quality. In contrast, CRQS reveals clinically meaningful differences between models and encoders that lexical metrics fail to capture. Together, the benchmark, public plug-and-play framework, and CRQS establish a reproducible foundation for rigorous evaluation of pathology report generation.