# Is human scoring the best criteria for summary evaluation?

arXiv: https://arxiv.org/abs/2012.14602 · HTML: https://mlsanity.com/paper/2012.14602/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Dec 2020.

## Built on by (most cited first, 7 of 7)

- [A Survey on Multi-modal Summarization](https://mlsanity.com/paper/2109.05199/) (2021)
- [Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings](https://mlsanity.com/paper/2104.05156/) (2021)
- [Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics](https://mlsanity.com/paper/2410.10867/) (2024)
- [WIDAR -- Weighted Input Document Augmented ROUGE](https://mlsanity.com/paper/2201.09282/) (2022)
- [Does Summary Evaluation Survive Translation to Other Languages?](https://mlsanity.com/paper/2109.08129/) (2021)
- [Consistency and Coherence from Points of Contextual Similarity](https://mlsanity.com/paper/2112.11638/) (2021)
- [Neural Embeddings for Text](https://mlsanity.com/paper/2208.08386/) (2022)

## Abstract (arXiv)

Normally, summary quality measures are compared with quality scores produced by human annotators. A higher correlation with human scores is considered to be a fair indicator of a better measure. We discuss observations that cast doubt on this view. We attempt to show a possibility of an alternative indicator. Given a family of measures, we explore a criterion of selecting the best measure not relying on correlations with human scores. Our observations for the BLANC family of measures suggest that the criterion is universal across very different styles of summaries.