# Efficient Adaptation of Pretrained Transformers for Abstractive Summarization

arXiv: https://arxiv.org/abs/1906.00138 · HTML: https://mlsanity.com/paper/1906.00138/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2019.

## Built on by (most cited first, 13 of 13)

- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Towards Label-Only Membership Inference Attack against Pre-trained Large Language Models](https://mlsanity.com/paper/2502.18943/) (2025)
- [Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report](https://mlsanity.com/paper/2504.21039/) (2025)
- [Nutri-bullets: Summarizing Health Studies by Composing Segments](https://mlsanity.com/paper/2103.11921/) (2021)
- [BC4LLM: Trusted Artificial Intelligence When Blockchain Meets Large Language Models](https://mlsanity.com/paper/2310.06278/) (2023)
- [Tag&Tab: Pretraining Data Detection in Large Language Models Using Keyword-Based Membership Inference Attack](https://mlsanity.com/paper/2501.08454/) (2025)
- [Nutribullets Hybrid: Multi-document Health Summarization](https://mlsanity.com/paper/2104.03465/) (2021)
- [Hashtag-Guided Low-Resource Tweet Classification](https://mlsanity.com/paper/2302.10143/) (2023)
- [Unintended Memorization and Timing Attacks in Named Entity Recognition Models](https://mlsanity.com/paper/2211.02245/) (2022)
- [LexiMark: Robust Watermarking via Lexical Substitutions to Enhance Membership Verification of an LLM's Textual Training Data](https://mlsanity.com/paper/2506.14474/) (2025)
- [Sample Efficient Multimodal Semantic Augmentation for Incremental Summarization](https://mlsanity.com/paper/2303.04361/) (2023)

## Abstract (arXiv)

Large-scale learning of transformer language models has yielded improvements on a variety of natural language understanding tasks. Whether they can be effectively adapted for summarization, however, has been less explored, as the learned representations are less seamlessly integrated into existing neural text production architectures. In this work, we propose two solutions for efficiently adapting pretrained transformer language models as text summarizers: source embeddings and domain-adaptive training. We test these solutions on three abstractive summarization datasets, achieving new state of the art performance on two of them. Finally, we show that these improvements are achieved by producing more focused summaries with fewer superfluous and that performance improvements are more pronounced on more abstractive datasets.