# Evaluating the Efficacy of Summarization Evaluation across Languages

arXiv: https://arxiv.org/abs/2106.01478 · HTML: https://mlsanity.com/paper/2106.01478/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization](https://mlsanity.com/paper/2010.03093/) (2020)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [MLSUM: The Multilingual Summarization Corpus](https://mlsanity.com/paper/2004.14900/) (2020)
- [Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language](https://mlsanity.com/paper/1905.07213/) (2019)
- [LCSTS: A Large Scale Chinese Short Text Summarization Dataset](https://mlsanity.com/paper/1506.05865/) (2015)
- [IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP](https://mlsanity.com/paper/2011.00677/) (2020)
- [Unsupervised Opinion Summarization with Noising and Denoising](https://mlsanity.com/paper/2004.10150/) (2020)
- [Revisiting Summarization Evaluation for Scientific Articles](https://mlsanity.com/paper/1604.00400/) (2016)
- [How Furiously Can Colourless Green Ideas Sleep? Sentence Acceptability in Context](https://mlsanity.com/paper/2004.00881/) (2020)
- [FFCI: A Framework for Interpretable Automatic Evaluation of Summarization](https://mlsanity.com/paper/2011.13662/) (2020)
- [HighRES: Highlight-based Reference-less Evaluation of Summarization](https://mlsanity.com/paper/1906.01361/) (2019)
- [Liputan6: A Large-scale Indonesian Dataset for Text Summarization](https://mlsanity.com/paper/2011.00679/) (2020)

## Built on by (most cited first, 14 of 14)

- [Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation](https://mlsanity.com/paper/2205.12647/) (2022)
- [SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization](https://mlsanity.com/paper/2203.06569/) (2022)
- [A Survey on Cross-Lingual Summarization](https://mlsanity.com/paper/2203.12515/) (2022)
- [CaLMQA: Exploring culturally specific long-form question answering across 23 languages](https://mlsanity.com/paper/2406.17761/) (2024)
- [T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics](https://mlsanity.com/paper/2212.05726/) (2022)
- [BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation](https://mlsanity.com/paper/2110.09147/) (2021)
- [MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification](https://mlsanity.com/paper/2205.12394/) (2022)
- [Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization](https://mlsanity.com/paper/2507.08342/) (2025)
- [NusaCrowd: A Call for Open and Reproducible NLP Research in Indonesian Languages](https://mlsanity.com/paper/2207.10524/) (2022)
- [Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation](https://mlsanity.com/paper/2210.03018/) (2022)
- [Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English](https://mlsanity.com/paper/2503.17039/) (2025)
- [SumTra: A Differentiable Pipeline for Few-Shot Cross-Lingual Summarization](https://mlsanity.com/paper/2403.13240/) (2024)
- [Evaluating the Efficacy of Length-Controllable Machine Translation](https://mlsanity.com/paper/2305.02300/) (2023)
- [Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages](https://mlsanity.com/paper/2510.07061/) (2025)

## Abstract (arXiv)

While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first attempt to systematically quantify their panlinguistic efficacy. We take a summarization corpus for eight different languages, and manually annotate generated summaries for focus (precision) and coverage (recall). Based on this, we evaluate 19 summarization evaluation metrics, and find that using multilingual BERT within BERTScore performs well across all languages, at a level above that for English.