# Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization

arXiv: https://arxiv.org/abs/2108.02859 · HTML: https://mlsanity.com/paper/2108.02859/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 5 Aug 2021.

## Builds on (most cited first, 30 of 35)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [Neural Text Summarization: A Critical Evaluation](https://mlsanity.com/paper/1908.08960/) (2019)
- [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017)
- [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021)
- [Evaluating Factuality in Generation with Dependency-level Entailment](https://mlsanity.com/paper/2010.05478/) (2020)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)
- [Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports](https://mlsanity.com/paper/1911.02541/) (2019)
- [Answers Unite! Unsupervised Metrics for Reinforced Summarization Models](https://mlsanity.com/paper/1909.01610/) (2019)
- [GO FIGURE: A Meta Evaluation of Factuality in Summarization](https://mlsanity.com/paper/2010.12834/) (2020)
- [Pointing the Unknown Words](https://mlsanity.com/paper/1603.08148/) (2016)
- [Improving Abstraction in Text Summarization](https://mlsanity.com/paper/1808.07913/) (2018)
- [Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR](https://mlsanity.com/paper/2008.08896/) (2020)
- [Analyzing Sentence Fusion in Abstractive Summarization](https://mlsanity.com/paper/1910.00203/) (2019)

## Built on by (most cited first, 16 of 16)

- [Faithfulness-Aware Decoding Strategies for Abstractive Summarization](https://mlsanity.com/paper/2303.03278/) (2023)
- [Generating Summaries with Controllable Readability Levels](https://mlsanity.com/paper/2310.10623/) (2023)
- [From Chaos to Clarity: Claim Normalization to Empower Fact-Checking](https://mlsanity.com/paper/2310.14338/) (2023)
- [Enhancing Abstractiveness of Summarization Models through Calibrated Distillation](https://mlsanity.com/paper/2310.13760/) (2023)
- [Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality](https://mlsanity.com/paper/2305.14981/) (2023)
- [Model-based Preference Optimization in Abstractive Summarization without Human Feedback](https://mlsanity.com/paper/2409.18618/) (2024)
- [Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection](https://mlsanity.com/paper/2403.04073/) (2024)
- [Utilizing GPT to Enhance Text Summarization: A Strategy to Minimize Hallucinations](https://mlsanity.com/paper/2405.04039/) (2024)
- [Disentangling Likes and Dislikes in Personalized Generative Explainable Recommendation](https://mlsanity.com/paper/2410.13248/) (2024)
- [Reranking-based Generation for Unbiased Perspective Summarization](https://mlsanity.com/paper/2506.15925/) (2025)
- [The Extractive-Abstractive Axis: Measuring Content "Borrowing" in Generative Language Models](https://mlsanity.com/paper/2307.11779/) (2023)
- [Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue](https://mlsanity.com/paper/2402.02077/) (2024)
- [Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models](https://mlsanity.com/paper/2402.15162/) (2024)
- [MMD-Flagger: Leveraging Maximum Mean Discrepancy to Detect Hallucinations](https://mlsanity.com/paper/2506.01367/) (2025)
- [No Reader Left Behind: Multi-Agent Summaries Everyone Can Understand](https://mlsanity.com/paper/2605.28836/) (2026)
- [Copying Is Not Grounding: Training Faithful Long-Form RAG with Authored-Sentence Credit](https://mlsanity.com/paper/2610.18442/) (2026)

## Abstract (arXiv)

Neural models for abstractive summarization tend to generate output that is fluent and well-formed but lacks semantic faithfulness, or factuality, with respect to the input documents. In this paper, we analyze the tradeoff between abstractiveness and factuality of generated summaries across multiple datasets and models, using extensive human evaluations of factuality. In our analysis, we visualize the rates of change in factuality as we gradually increase abstractiveness using a decoding constraint, and we observe that, while increased abstractiveness generally leads to a drop in factuality, the rate of factuality decay depends on factors such as the data that the system was trained on. We introduce two datasets with human factuality judgements; one containing 10.2k generated summaries with systematically varied degrees of abstractiveness; the other containing 4.2k summaries from five different summarization models. We propose new factuality metrics that adjust for the degree of abstractiveness, and we use them to compare the abstractiveness-adjusted factuality of previous summarization works, providing baselines for future work.