mlsanityData to 14 Oct 2026

Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings

arXiv 2104.05156 ↗First posted 12 Apr 2021cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We propose a new reference-free summary quality evaluation measure, with emphasis on the faithfulness. The measure is designed to find and count all possible minute inconsistencies of the summary with respect to the source document. The proposed ESTIME, Estimator of Summary-to-Text Inconsistency by Mismatched Embeddings, correlates with expert scores in summary-level SummEval dataset stronger than other common evaluation measures not only in Consistency but also in Fluency. We also introduce a method of generating subtle factual errors in human summaries. We show that ESTIME is more sensitive to subtle errors than other common evaluation measures.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 24 of its 29 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

24 of 29 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2015Teaching Machines to Read and Comprehendcited by 1,049
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  5. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  7. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  8. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  9. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  10. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  11. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  12. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
12 more
  1. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  2. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
  3. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 82
  4. 2020GO FIGURE: A Meta Evaluation of Factuality in Summarizationcited by 76
  5. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  6. 2020What Have We Achieved on Text Summarization?cited by 65
  7. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  8. 2019SumQE: a BERT-based Summary Quality Estimation Modelcited by 17
  9. 2021Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluationcited by 13
  10. 2018Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Informationcited by 8
  11. 2020Is human scoring the best criteria for summary evaluation?cited by 7
  12. 2020Sensitivity of BLANC to human-scored qualities of text summariescited by 6

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 0; 2023: 1; 2024: 1; 2025: 1.

5 papers.

  1. 2023Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Studycited by 5
  2. 2024Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metricscited by 5
  3. 2021Does Summary Evaluation Survive Translation to Other Languages?cited by 1
  4. 2021Consistency and Coherence from Points of Contextual Similaritycited by 1
  5. 2025Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structurecited by 0