# Exploring Multitask Learning for Low-Resource AbstractiveSummarization

arXiv: https://arxiv.org/abs/2109.08565 · HTML: https://mlsanity.com/paper/2109.08565/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018)
- [AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization](https://mlsanity.com/paper/2103.11332/) (2021)
- [A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss](https://mlsanity.com/paper/1805.06266/) (2018)
- [Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation](https://mlsanity.com/paper/1805.11004/) (2018)
- [A Unified Dual-view Model for Review Summarization and Sentiment Classification with Inconsistency Loss](https://mlsanity.com/paper/2006.01592/) (2020)
- [Abstractive Summarization for Low Resource Data using Domain Transfer and Data Synthesis](https://mlsanity.com/paper/2002.03407/) (2020)
- [A Hierarchical End-to-End Model for Jointly Improving Text Summarization and Sentiment Classification](https://mlsanity.com/paper/1805.01089/) (2018)
- [Attend to the beginning: A study on using bidirectional attention for extractive summarization](https://mlsanity.com/paper/2002.03405/) (2020)

## Built on by (most cited first, 5 of 5)

- [Explaining Veracity Predictions with Evidence Summarization: A Multi-Task Model Approach](https://mlsanity.com/paper/2402.06443/) (2024)
- [Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning](https://mlsanity.com/paper/2305.12077/) (2023)
- [Leveraging Key Information Modeling to Improve Less-Data Constrained News Headline Generation via Duality Fine-Tuning](https://mlsanity.com/paper/2210.04473/) (2022)
- [SPEC: Summary Preference Decomposition for Low-Resource Abstractive Summarization](https://mlsanity.com/paper/2303.14011/) (2023)
- [ReflectSumm: A Benchmark for Course Reflection Summarization](https://mlsanity.com/paper/2403.19012/) (2024)

## Abstract (arXiv)

This paper explores the effect of using multitask learning for abstractive summarization in the context of small training corpora. In particular, we incorporate four different tasks (extractive summarization, language modeling, concept detection, and paraphrase detection) both individually and in combination, with the goal of enhancing the target task of abstractive summarization via multitask learning. We show that for many task combinations, a model trained in a multitask setting outperforms a model trained only for abstractive summarization, with no additional summarization data introduced. Additionally, we do a comprehensive search and find that certain tasks (e.g. paraphrase detection) consistently benefit abstractive summarization, not only when combined with other tasks but also when using different architectures and training corpora.