# Evaluation of Automatic Text Summarization using Synthetic Facts

arXiv: https://arxiv.org/abs/2204.04869 · HTML: https://mlsanity.com/paper/2204.04869/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 15 of 15)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [Detecting Hallucinated Content in Conditional Neural Sequence Generation](https://mlsanity.com/paper/2011.02593/) (2020)
- [Evaluating Factuality in Generation with Dependency-level Entailment](https://mlsanity.com/paper/2010.05478/) (2020)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)
- [Answers Unite! Unsupervised Metrics for Reinforced Summarization Models](https://mlsanity.com/paper/1909.01610/) (2019)
- [Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary](https://mlsanity.com/paper/2010.00490/) (2020)
- [Content Selection in Deep Learning Models of Summarization](https://mlsanity.com/paper/1810.12343/) (2018)
- [Question Answering as an Automatic Evaluation Metric for News Article Summarization](https://mlsanity.com/paper/1906.00318/) (2019)
- [Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning](https://mlsanity.com/paper/2010.01781/) (2020)
- [Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation](https://mlsanity.com/paper/2103.10918/) (2021)

## Abstract (arXiv)

Despite some recent advances, automatic text summarization remains unreliable, elusive, and of limited practical use in applications. Two main problems with current summarization methods are well known: evaluation and factual consistency. To address these issues, we propose a new automatic reference-less text summarization evaluation system that can measure the quality of any text summarization model with a set of generated facts based on factual consistency, comprehensiveness, and compression rate. As far as we know, our evaluation system is the first system that measures the overarching quality of the text summarization models based on factuality, information coverage, and compression rate.