# Extract-and-Abstract: Unifying Extractive and Abstractive Summarization within Single Encoder-Decoder Framework

arXiv: https://arxiv.org/abs/2409.11827 · HTML: https://mlsanity.com/paper/2409.11827/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 28 of 28)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [Extractive Summarization as Text Matching](https://mlsanity.com/paper/2004.08795/) (2020)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [GSum: A General Framework for Guided Neural Abstractive Summarization](https://mlsanity.com/paper/2010.08014/) (2020)
- [Discourse-Aware Neural Extractive Text Summarization](https://mlsanity.com/paper/1910.14142/) (2019)
- [Abstractive Summarization of Reddit Posts with Multi-level Memory Networks](https://mlsanity.com/paper/1811.00783/) (2018)
- [On Extractive and Abstractive Neural Document Summarization with Transformer Language Models](https://mlsanity.com/paper/1909.03186/) (2019)
- [Extractive Summarization via ChatGPT for Faithful Summary Generation](https://mlsanity.com/paper/2304.04193/) (2023)
- [A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss](https://mlsanity.com/paper/1805.06266/) (2018)
- [HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information](https://mlsanity.com/paper/2203.09629/) (2022)
- [Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization](https://mlsanity.com/paper/2305.16784/) (2023)
- [Salience Allocation as Guidance for Abstractive Summarization](https://mlsanity.com/paper/2210.12330/) (2022)
- [Improving Multi-Document Summarization through Referenced Flexible Extraction with Credit-Awareness](https://mlsanity.com/paper/2205.01889/) (2022)
- [Jointly Extracting and Compressing Documents with Summary State Representations](https://mlsanity.com/paper/1904.02020/) (2019)
- [HydraSum: Disentangling Stylistic Features in Text Summarization using Multi-Decoder Models](https://mlsanity.com/paper/2110.04400/) (2021)
- [A Cascade Approach to Neural Abstractive Summarization with Content Selection and Fusion](https://mlsanity.com/paper/2010.03722/) (2020)
- [Generating EDU Extracts for Plan-Guided Summary Re-Ranking](https://mlsanity.com/paper/2305.17779/) (2023)
- [Contextualized Rewriting for Text Summarization](https://mlsanity.com/paper/2102.00385/) (2021)
- [EDU-level Extractive Summarization with Varying Summary Lengths](https://mlsanity.com/paper/2210.04029/) (2022)

## Abstract (arXiv)

Extract-then-Abstract is a naturally coherent paradigm to conduct abstractive summarization with the help of salient information identified by the extractive model. Previous works that adopt this paradigm train the extractor and abstractor separately and introduce extra parameters to highlight the extracted salients to the abstractor, which results in error accumulation and additional training costs. In this paper, we first introduce a parameter-free highlight method into the encoder-decoder framework: replacing the encoder attention mask with a saliency mask in the cross-attention module to force the decoder to focus only on salient parts of the input. A preliminary analysis compares different highlight methods, demonstrating the effectiveness of our saliency mask. We further propose the novel extract-and-abstract paradigm, ExtAbs., which jointly and seamlessly performs Extractive and Abstractive summarization tasks within single encoder-decoder model to reduce error accumulation. In ExtAbs, the vanilla encoder is augmented to extract salients, and the vanilla decoder is modified with the proposed saliency mask to generate summaries. Built upon BART and PEGASUS, experiments on three datasets show that ExtAbs can achieve superior performance than baselines on the extractive task and performs comparable, or even better than the vanilla models on the abstractive task.