# Revisiting Automatic Question Summarization Evaluation in the Biomedical Domain

arXiv: https://arxiv.org/abs/2303.10328 · HTML: https://mlsanity.com/paper/2303.10328/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 18 Mar 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [Neural Text Summarization: A Critical Evaluation](https://mlsanity.com/paper/1908.08960/) (2019)
- [MedDialog: Two Large-scale Medical Dialogue Datasets](https://mlsanity.com/paper/2004.03329/) (2020)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://mlsanity.com/paper/2204.03905/) (2022)
- [Fill in the BLANC: Human-free quality estimation of document summaries](https://mlsanity.com/paper/2002.09836/) (2020)
- [A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task](https://mlsanity.com/paper/1606.02858/) (2016)
- [Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics](https://mlsanity.com/paper/2204.10216/) (2022)

## Built on by (most cited first, 1 of 1)

- [OpinSummEval: Revisiting Automated Evaluation for Opinion Summarization](https://mlsanity.com/paper/2310.18122/) (2023)

## Abstract (arXiv)

Automatic evaluation metrics have been facilitating the rapid development of automatic summarization methods by providing instant and fair assessments of the quality of summaries. Most metrics have been developed for the general domain, especially news and meeting notes, or other language-generation tasks. However, these metrics are applied to evaluate summarization systems in different domains, such as biomedical question summarization. To better understand whether commonly used evaluation metrics are capable of evaluating automatic summarization in the biomedical domain, we conduct human evaluations of summarization quality from four different aspects of a biomedical question summarization task. Based on human judgments, we identify different noteworthy features for current automatic metrics and summarization systems as well. We also release a dataset of our human annotations to aid the research of summarization evaluation metrics in the biomedical domain.