# Towards objectively evaluating the quality of generated medical summaries

arXiv: https://arxiv.org/abs/2104.04412 · HTML: https://mlsanity.com/paper/2104.04412/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 10 of 10)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Leveraging BERT for Extractive Text Summarization on Lectures](https://mlsanity.com/paper/1906.04165/) (2019)
- [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019)
- [Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference](https://mlsanity.com/paper/2011.10819/) (2020)
- [A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems](https://mlsanity.com/paper/2011.03992/) (2020)
- [ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversation](https://mlsanity.com/paper/1910.12129/) (2019)
- [Neural Latent Extractive Document Summarization](https://mlsanity.com/paper/1808.07187/) (2018)

## Built on by (most cited first, 5 of 5)

- [Summarizing Patients Problems from Hospital Progress Notes Using Pre-trained Sequence-to-Sequence Models](https://mlsanity.com/paper/2208.08408/) (2022)
- [A Survey on Medical Document Summarization](https://mlsanity.com/paper/2212.01669/) (2022)
- [A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization](https://mlsanity.com/paper/2303.03948/) (2023)
- [A Survey for Biomedical Text Summarization: From Pre-trained to Large Language Models](https://mlsanity.com/paper/2304.08763/) (2023)
- [Conical Classification For Computationally Efficient One-Class Topic Determination](https://mlsanity.com/paper/2111.00375/) (2021)

## Abstract (arXiv)

We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.