# Abstractive Summarization with Combination of Pre-trained Sequence-to-Sequence and Saliency Models

arXiv: https://arxiv.org/abs/2003.13028 · HTML: https://mlsanity.com/paper/2003.13028/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2020.

## Built on by (most cited first, 15 of 15)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods](https://mlsanity.com/paper/2203.05227/) (2022)
- [EntSUM: A Data Set for Entity-Centric Summarization](https://mlsanity.com/paper/2204.02213/) (2022)
- [RetrievalSum: A Retrieval Enhanced Framework for Abstractive Summarization](https://mlsanity.com/paper/2109.07943/) (2021)
- [Salience Allocation as Guidance for Abstractive Summarization](https://mlsanity.com/paper/2210.12330/) (2022)
- [Leveraging Information Bottleneck for Scientific Document Summarization](https://mlsanity.com/paper/2110.01280/) (2021)
- [Generating EDU Extracts for Plan-Guided Summary Re-Ranking](https://mlsanity.com/paper/2305.17779/) (2023)
- [Prompt-based Learning for Text Readability Assessment](https://mlsanity.com/paper/2302.13139/) (2023)
- [EASE: Extractive-Abstractive Summarization with Explanations](https://mlsanity.com/paper/2105.06982/) (2021)
- [Controlling the Focus of Pretrained Language Generation Models](https://mlsanity.com/paper/2203.01146/) (2022)
- [A General Contextualized Rewriting Framework for Text Summarization](https://mlsanity.com/paper/2207.05948/) (2022)
- [Analysis of Multidomain Abstractive Summarization Using Salience Allocation](https://mlsanity.com/paper/2402.11955/) (2024)
- [EROS: Entity-Driven Controlled Policy Document Summarization](https://mlsanity.com/paper/2403.00141/) (2024)
- [ROUGE-K: Do Your Summaries Have Keywords?](https://mlsanity.com/paper/2403.05186/) (2024)

## Abstract (arXiv)

Pre-trained sequence-to-sequence (seq-to-seq) models have significantly improved the accuracy of several language generation tasks, including abstractive summarization. Although the fluency of abstractive summarization has been greatly improved by fine-tuning these models, it is not clear whether they can also identify the important parts of the source text to be included in the summary. In this study, we investigated the effectiveness of combining saliency models that identify the important parts of the source text with the pre-trained seq-to-seq models through extensive experiments. We also proposed a new combination model consisting of a saliency model that extracts a token sequence from a source text and a seq-to-seq model that takes the sequence as an additional input text. Experimental results showed that most of the combination models outperformed a simple fine-tuned seq-to-seq model on both the CNN/DM and XSum datasets even if the seq-to-seq model is pre-trained on large-scale corpora. Moreover, for the CNN/DM dataset, the proposed combination model exceeded the previous best-performed model by 1.33 points on ROUGE-L.