# Meta-Transfer Learning for Low-Resource Abstractive Summarization

arXiv: https://arxiv.org/abs/2102.09397 · HTML: https://mlsanity.com/paper/2102.09397/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 18 Feb 2021.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018)
- [BillSum: A Corpus for Automatic Summarization of US Legislation](https://mlsanity.com/paper/1910.00523/) (2019)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1808.08437/) (2018)
- [Abstractive Summarization of Reddit Posts with Multi-level Memory Networks](https://mlsanity.com/paper/1811.00783/) (2018)
- [Personalizing Dialogue Agents via Meta-Learning](https://mlsanity.com/paper/1905.10033/) (2019)
- [This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation](https://mlsanity.com/paper/1906.03497/) (2019)
- [Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks](https://mlsanity.com/paper/1908.10423/) (2019)
- [Domain Adaptive Dialog Generation via Meta Learning](https://mlsanity.com/paper/1906.03520/) (2019)
- [ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation](https://mlsanity.com/paper/2001.11314/) (2020)
- [Sample Efficient Text Summarization Using a Single Pre-Trained Transformer](https://mlsanity.com/paper/1905.08836/) (2019)
- [Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing](https://mlsanity.com/paper/1906.07108/) (2019)
- [MetaMT,a MetaLearning Method Leveraging Multiple Domain Data for Low Resource Machine Translation](https://mlsanity.com/paper/1912.05467/) (2019)

## Built on by (most cited first, 14 of 14)

- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [A Three-Stage Learning Framework for Low-Resource Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2109.04096/) (2021)
- [Boosting Natural Language Generation from Instructions with Meta-Learning](https://mlsanity.com/paper/2210.11617/) (2022)
- [UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization](https://mlsanity.com/paper/2211.09783/) (2022)
- [InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT](https://mlsanity.com/paper/2305.13083/) (2023)
- [SINC: Self-Supervised In-Context Learning for Vision-Language Tasks](https://mlsanity.com/paper/2307.07742/) (2023)
- [To Adapt or to Fine-tune: A Case Study on Abstractive Summarization](https://mlsanity.com/paper/2208.14559/) (2022)
- [Learning to Adapt to Low-Resource Paraphrase Generation](https://mlsanity.com/paper/2412.17111/) (2024)
- [From News to Returns: A Granger-Causal Hypergraph Transformer on the Sphere](https://mlsanity.com/paper/2510.04357/) (2025)
- [Cross-Domain Robustness of Transformer-based Keyphrase Generation](https://mlsanity.com/paper/2312.10700/) (2023)
- [Meta-RTL: Reinforcement-Based Meta-Transfer Learning for Low-Resource Commonsense Reasoning](https://mlsanity.com/paper/2409.19075/) (2024)
- [State Space Models for Extractive Summarization in Low Resource Scenarios](https://mlsanity.com/paper/2501.14673/) (2025)
- [MetaXCR: Reinforcement-Based Meta-Transfer Learning for Cross-Lingual Commonsense Reasoning](https://mlsanity.com/paper/2503.06531/) (2025)
- [A Split-then-Join Approach to Abstractive Summarization for Very Long Documents in a Low Resource Setting](https://mlsanity.com/paper/2505.06862/) (2025)

## Abstract (arXiv)

Neural abstractive summarization has been studied in many pieces of literature and achieves great success with the aid of large corpora. However, when encountering novel tasks, one may not always benefit from transfer learning due to the domain shifting problem, and overfitting could happen without adequate labeled examples. Furthermore, the annotations of abstractive summarization are costly, which often demand domain knowledge to ensure the ground-truth quality. Thus, there are growing appeals for Low-Resource Abstractive Summarization, which aims to leverage past experience to improve the performance with limited labeled examples of target corpus. In this paper, we propose to utilize two knowledge-rich sources to tackle this problem, which are large pre-trained models and diverse existing corpora. The former can provide the primary ability to tackle summarization tasks; the latter can help discover common syntactic or semantic information to improve the generalization ability. We conduct extensive experiments on various summarization corpora with different writing styles and forms. The results demonstrate that our approach achieves the state-of-the-art on 6 corpora in low-resource scenarios, with only 0.7% of trainable parameters compared to previous work.