mlsanityData to 14 Oct 2026

Towards Human-Free Automatic Quality Evaluation of German Summarization

arXiv 2105.06027 ↗First posted 13 May 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Evaluating large summarization corpora using humans has proven to be expensive from both the organizational and the financial perspective. Therefore, many automatic evaluation metrics have been developed to measure the summarization quality in a fast and reproducible way. However, most of the metrics still rely on humans and need gold standard summaries generated by linguistic experts. Since BLANC does not require golden summaries and supposedly can use any underlying language model, we consider its application to the evaluation of summarization in German. This work demonstrates how to adjust the BLANC metric to a language other than English. We compare BLANC scores with the crowd and expert ratings, as well as with commonly used automatic metrics on a German summarization data set. Our results show that BLANC in German is especially good in evaluating informativeness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 33 references that are in the map. First posted on arXiv 13 May 2021.

Builds on

15 of 33 references.

References · 15

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  3. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  4. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  5. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  6. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  7. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  8. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 82
  9. 2020GO FIGURE: A Meta Evaluation of Factuality in Summarizationcited by 76
  10. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  11. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  12. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
3 more
  1. 2019Crowdsourcing Lightweight Pyramids for Manual Summary Evaluationcited by 39
  2. 2019SumQE: a BERT-based Summary Quality Estimation Modelcited by 17
  3. 2020Sensitivity of BLANC to human-scored qualities of text summariescited by 6

Built on by

Not cited yet.