# Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization

arXiv: https://arxiv.org/abs/2209.03549 · HTML: https://mlsanity.com/paper/2209.03549/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 77 references that are in the map.
- First posted on arXiv 8 Sep 2022.

## Builds on (most cited first, 30 of 46)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [Extractive Summarization as Text Matching](https://mlsanity.com/paper/2004.08795/) (2020)
- [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017)
- [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021)
- [Ranking Sentences for Extractive Summarization with Reinforcement Learning](https://mlsanity.com/paper/1802.08636/) (2018)
- [Evaluating Factuality in Generation with Dependency-level Entailment](https://mlsanity.com/paper/2010.05478/) (2020)
- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2109.09209/) (2021)
- [Discourse-Aware Neural Extractive Text Summarization](https://mlsanity.com/paper/1910.14142/) (2019)
- [Decontextualization: Making Sentences Stand-Alone](https://mlsanity.com/paper/2102.05169/) (2021)
- [Heterogeneous Graph Neural Networks for Extractive Document Summarization](https://mlsanity.com/paper/2004.12393/) (2020)

## Built on by (most cited first, 30 of 32)

- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [Evaluating the Factual Consistency of Large Language Models Through News Summarization](https://mlsanity.com/paper/2211.08412/) (2022)
- [DocFinQA: A Long-Context Financial Reasoning Dataset](https://mlsanity.com/paper/2401.06915/) (2024)
- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023)
- [Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization](https://mlsanity.com/paper/2407.12848/) (2024)
- [SEMQA: Semi-Extractive Multi-Source Question Answering](https://mlsanity.com/paper/2311.04886/) (2023)
- [A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documents](https://mlsanity.com/paper/2305.14772/) (2023)
- [A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization](https://mlsanity.com/paper/2303.03948/) (2023)
- [Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations](https://mlsanity.com/paper/2402.05629/) (2024)
- [An Investigation of Evaluation Metrics for Automated Medical Note Generation](https://mlsanity.com/paper/2305.17364/) (2023)
- [Context-Aware Hierarchical Merging for Long Document Summarization](https://mlsanity.com/paper/2502.00977/) (2025)
- [What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization](https://mlsanity.com/paper/2305.07615/) (2023)
- [Beyond Borders: Investigating Cross-Jurisdiction Transfer in Legal Case Summarization](https://mlsanity.com/paper/2403.19317/) (2024)
- [Concise Answers to Complex Questions: Summarization of Long-form Answers](https://mlsanity.com/paper/2305.19271/) (2023)
- [A Survey of Automatic Hallucination Evaluation on Natural Language Generation](https://mlsanity.com/paper/2404.12041/) (2024)
- [StrucSum: Graph-Structured Reasoning for Long Document Extractive Summarization with LLMs](https://mlsanity.com/paper/2505.22950/) (2025)
- [Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agents](https://mlsanity.com/paper/2605.08580/) (2026)
- [Incremental Extractive Opinion Summarization Using Cover Trees](https://mlsanity.com/paper/2401.08047/) (2024)
- [CoPERLex: Content Planning with Event-based Representations for Legal Case Summarization](https://mlsanity.com/paper/2501.14112/) (2025)
- [DTCRS: Dynamic Tree Construction for Recursive Summarization](https://mlsanity.com/paper/2604.07012/) (2026)
- [STRONG -- Structure Controllable Legal Opinion Summary Generation](https://mlsanity.com/paper/2309.17280/) (2023)
- [FREDSum: A Dialogue Summarization Corpus for French Political Debates](https://mlsanity.com/paper/2312.04843/) (2023)
- [Leveraging Discourse Structure for Extractive Meeting Summarization](https://mlsanity.com/paper/2405.11055/) (2024)
- [SKETCH: Structured Knowledge Enhanced Text Comprehension for Holistic Retrieval](https://mlsanity.com/paper/2412.15443/) (2024)
- [Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries](https://mlsanity.com/paper/2505.21859/) (2025)
- [Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research](https://mlsanity.com/paper/2608.24306/) (2026)
- [Unsupervised Extractive Dialogue Summarization in Hyperdimensional Space](https://mlsanity.com/paper/2405.09765/) (2024)
- [PECAN: LLM-Guided Dynamic Progress Control with Attention-Guided Hierarchical Weighted Graph for Long-Document QA](https://mlsanity.com/paper/2410.04790/) (2024)
- [Improving Zero-shot Sentence Decontextualisation with Content Selection and Planning](https://mlsanity.com/paper/2509.17921/) (2025)

## Abstract (arXiv)

The problems of unfaithful summaries have been widely discussed under the context of abstractive summarization. Though extractive summarization is less prone to the common unfaithfulness issues of abstractive summaries, does that mean extractive is equal to faithful? Turns out that the answer is no. In this work, we define a typology with five types of broad unfaithfulness problems (including and beyond not-entailment) that can appear in extractive summaries, including incorrect coreference, incomplete coreference, incorrect discourse, incomplete discourse, as well as other misleading information. We ask humans to label these problems out of 1600 English summaries produced by 16 diverse extractive systems. We find that 30% of the summaries have at least one of the five issues. To automatically detect these problems, we find that 5 existing faithfulness evaluation metrics for summarization have poor correlations with human judgment. To remedy this, we propose a new metric, ExtEval, that is designed for detecting unfaithful extractive summaries and is shown to have the best performance. We hope our work can increase the awareness of unfaithfulness problems in extractive summarization and help future work to evaluate and resolve these issues. Our data and code are publicly available at https://github.com/ZhangShiyue/extractive_is_not_faithful