# Sample Efficient Text Summarization Using a Single Pre-Trained Transformer

arXiv: https://arxiv.org/abs/1905.08836 · HTML: https://mlsanity.com/paper/1905.08836/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2019.

## Built on by (most cited first, 27 of 27)

- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [Planning with Learned Entity Prompts for Abstractive Summarization](https://mlsanity.com/paper/2104.07606/) (2021)
- [Uncertainty Estimation for Language Reward Models](https://mlsanity.com/paper/2203.07472/) (2022)
- [Symbolic Behaviour in Artificial Intelligence](https://mlsanity.com/paper/2102.03406/) (2021)
- [Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?](https://mlsanity.com/paper/2308.00189/) (2023)
- [See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization](https://mlsanity.com/paper/2105.09601/) (2021)
- [Meta-Transfer Learning for Low-Resource Abstractive Summarization](https://mlsanity.com/paper/2102.09397/) (2021)
- [Reasoning Elicitation in Language Models via Counterfactual Feedback](https://mlsanity.com/paper/2410.03767/) (2024)
- [Leveraging Information Bottleneck for Scientific Document Summarization](https://mlsanity.com/paper/2110.01280/) (2021)
- [A Frustratingly Simple Approach for End-to-End Image Captioning](https://mlsanity.com/paper/2201.12723/) (2022)
- [Efficient GPT Model Pre-training using Tensor Train Matrix Representation](https://mlsanity.com/paper/2306.02697/) (2023)
- [Fruit Ripeness Classification: a Survey](https://mlsanity.com/paper/2212.14441/) (2022)
- [Core Building Blocks: Next Gen Geo Spatial GPT Application](https://mlsanity.com/paper/2310.11029/) (2023)
- [DialogSum Challenge: Results of the Dialogue Summarization Shared Task](https://mlsanity.com/paper/2208.03898/) (2022)
- [vec2text with Round-Trip Translations](https://mlsanity.com/paper/2209.06792/) (2022)
- [Hashtag-Guided Low-Resource Tweet Classification](https://mlsanity.com/paper/2302.10143/) (2023)
- [Optimal path for Biomedical Text Summarization Using Pointer GPT](https://mlsanity.com/paper/2404.08654/) (2024)
- [Selective Token Generation for Few-shot Natural Language Generation](https://mlsanity.com/paper/2209.08206/) (2022)
- [SPEC: Summary Preference Decomposition for Low-Resource Abstractive Summarization](https://mlsanity.com/paper/2303.14011/) (2023)
- [Advancements in Natural Language Processing for Automatic Text Summarization](https://mlsanity.com/paper/2502.19773/) (2025)
- [Evaluating book summaries from internal knowledge in Large Language Models: a cross-model and semantic consistency approach](https://mlsanity.com/paper/2503.21613/) (2025)
- [Generating Rich Product Descriptions for Conversational E-commerce Systems](https://mlsanity.com/paper/2111.15298/) (2021)
- [Brand Celebrity Matching Model Based on Natural Language Processing](https://mlsanity.com/paper/2208.08887/) (2022)
- [Clinical Reading Comprehension with Encoder-Decoder Models Enhanced by Direct Preference Optimization](https://mlsanity.com/paper/2407.14000/) (2024)
- [FullTransNet: Full Transformer with Local-Global Attention for Video Summarization](https://mlsanity.com/paper/2501.00882/) (2025)

## Abstract (arXiv)

Language model (LM) pre-training has resulted in impressive performance and sample efficiency on a variety of language understanding tasks. However, it remains unclear how to best use pre-trained LMs for generation tasks such as abstractive summarization, particularly to enhance sample efficiency. In these sequence-to-sequence settings, prior work has experimented with loading pre-trained weights into the encoder and/or decoder networks, but used non-pre-trained encoder-decoder attention weights. We instead use a pre-trained decoder-only network, where the same Transformer LM both encodes the source and generates the summary. This ensures that all parameters in the network, including those governing attention over source states, have been pre-trained before the fine-tuning step. Experiments on the CNN/Daily Mail dataset show that our pre-trained Transformer LM substantially improves over pre-trained Transformer encoder-decoder networks in limited-data settings. For instance, it achieves 13.1 ROUGE-2 using only 1% of the training data (~3000 examples), while pre-trained encoder-decoder models score 2.3 ROUGE-2.