mlsanityData to 7 Oct 2026

Evaluating the Efficacy of Summarization Evaluation across Languages

arXiv 2106.01478 ↗First posted 2 Jun 2021cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first attempt to systematically quantify their panlinguistic efficacy. We take a summarization corpus for eight different languages, and manually annotate generated summaries for focus (precision) and coverage (recall). Based on this, we evaluate 19 summarization evaluation metrics, and find that using multilingual BERT within BERTScore performs well across all languages, at a level above that for English.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

26 of 41 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  6. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2019Text Summarization with Pretrained Encoderscited by 553
  11. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  12. 2019CamemBERT: a Tasty French Language Modelcited by 368
14 more
  1. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  2. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  3. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  4. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  5. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  6. 2019Adaptation of Deep Bidirectional Multilingual Transformers for Russian Languagecited by 109
  7. 2015LCSTS: A Large Scale Chinese Short Text Summarization Datasetcited by 65
  8. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 57
  9. 2020Unsupervised Opinion Summarization with Noising and Denoisingcited by 37
  10. 2016Revisiting Summarization Evaluation for Scientific Articlescited by 31
  11. 2020How Furiously Can Colourless Green Ideas Sleep? Sentence Acceptability in Contextcited by 26
  12. 2020FFCI: A Framework for Interpretable Automatic Evaluation of Summarizationcited by 24
  13. 2019HighRES: Highlight-based Reference-less Evaluation of Summarizationcited by 21
  14. 2020Liputan6: A Large-scale Indonesian Dataset for Text Summarizationcited by 14

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 7; 2023: 1; 2024: 2; 2025: 3.

14 papers.

Sort:

By area:

  1. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  2. 2022SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarizationcited by 61
  3. 2022A Survey on Cross-Lingual Summarizationcited by 38
  4. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  5. 2022T5Score: Discriminative Fine-tuning of Generative Evaluation Metricscited by 18
  6. 2021BEAMetrics: A Benchmark for Language Generation Evaluation Evaluationcited by 6
  7. 2022MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplificationcited by 3
  8. 2025Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarizationcited by 3
  9. 2022NusaCrowd: A Call for Open and Reproducible NLP Research in Indonesian Languagescited by 2
  10. 2022Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representationcited by 2
  11. 2025Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond Englishcited by 2
  12. 2024SumTra: A Differentiable Pipeline for Few-Shot Cross-Lingual Summarizationcited by 1
2 more
  1. 2023Evaluating the Efficacy of Length-Controllable Machine Translationcited by 0
  2. 2025Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languagescited by 0