mlsanityData to 7 Oct 2026

Does Summary Evaluation Survive Translation to Other Languages?

arXiv 2109.08129 ↗First posted 16 Sep 2021cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. If such effort is made in one language, it would be beneficial to be able to use it in other languages without repeating human annotations. To investigate how much we can trust machine translation of such a dataset, we translate the English dataset SummEval to seven languages and compare performance across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries. While we find some potential for dataset reuse in languages similar to the source, most summary evaluation methods are not found to be statistically equivalent across translations.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 28 references that are in the map. First posted on arXiv 16 Sep 2021.

Builds on

15 of 28 references.

References · 15

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2015Teaching Machines to Read and Comprehendcited by 1,039
  3. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  4. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  5. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  6. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  7. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 80
  8. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  9. 2021Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translationcited by 70
  10. 2019Translationese in Machine Translation Evaluationcited by 38
  11. 2020Curious Case of Language Generation Evaluation Metrics: A Cautionary Talecited by 28
  12. 2021Translation Error Detection as Rationale Extractioncited by 22
3 more
  1. 2021Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluationcited by 12
  2. 2020Is human scoring the best criteria for summary evaluation?cited by 7
  3. 2021Estimation of Summary-to-Text Inconsistency by Mismatched Embeddingscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154