# The Summary Loop: Learning to Write Abstractive Summaries Without Examples

arXiv: https://arxiv.org/abs/2105.05361 · HTML: https://mlsanity.com/paper/2105.05361/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 11 May 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports](https://mlsanity.com/paper/1911.02541/) (2019)
- [Answers Unite! Unsupervised Metrics for Reinforced Summarization Models](https://mlsanity.com/paper/1909.01610/) (2019)
- [Multi-Reward Reinforced Summarization with Saliency and Entailment](https://mlsanity.com/paper/1804.06451/) (2018)
- [Question Answering as an Automatic Evaluation Metric for News Article Summarization](https://mlsanity.com/paper/1906.00318/) (2019)
- [Cross-Lingual Natural Language Generation via Pre-Training](https://mlsanity.com/paper/1909.10481/) (2019)
- [BottleSum: Unsupervised and Self-supervised Sentence Summarization using the Information Bottleneck Principle](https://mlsanity.com/paper/1909.07405/) (2019)
- [Pre-trained Language Model Representations for Language Generation](https://mlsanity.com/paper/1903.09722/) (2019)
- [Variations of the Similarity Function of TextRank for Automated Summarization](https://mlsanity.com/paper/1602.03606/) (2016)
- [Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation](https://mlsanity.com/paper/1805.11004/) (2018)
- [Concept Pointer Network for Abstractive Summarization](https://mlsanity.com/paper/1910.08486/) (2019)
- [Reinforced Extractive Summarization with Question-Focused Rewards](https://mlsanity.com/paper/1805.10392/) (2018)
- [Abstractive Document Summarization without Parallel Data](https://mlsanity.com/paper/1907.12951/) (2019)
- [Attention Optimization for Abstractive Document Summarization](https://mlsanity.com/paper/1910.11491/) (2019)

## Built on by (most cited first, 27 of 27)

- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [Keep it Simple: Unsupervised Simplification of Multi-Paragraph Text](https://mlsanity.com/paper/2107.03444/) (2021)
- [Discrete Prompt Compression with Reinforcement Learning](https://mlsanity.com/paper/2308.08758/) (2023)
- [ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining](https://mlsanity.com/paper/2106.00829/) (2021)
- [Learning Opinion Summarizers by Selecting Informative Reviews](https://mlsanity.com/paper/2109.04325/) (2021)
- [Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewards](https://mlsanity.com/paper/2107.00176/) (2021)
- [Why is constrained neural language generation particularly challenging?](https://mlsanity.com/paper/2206.05395/) (2022)
- [Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning](https://mlsanity.com/paper/2205.08221/) (2022)
- [Discharge Summary Hospital Course Summarisation of In Patient Electronic Health Record Text with Clinical Concept Guided Deep Pre-Trained Transformer Models](https://mlsanity.com/paper/2211.07126/) (2022)
- [Controllable Summarization with Constrained Markov Decision Process](https://mlsanity.com/paper/2108.03405/) (2021)
- [DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4](https://mlsanity.com/paper/2305.14702/) (2023)
- [What's The Latest? A Question-driven News Chatbot](https://mlsanity.com/paper/2105.05392/) (2021)
- [Question-aware Transformer Models for Consumer Health Question Summarization](https://mlsanity.com/paper/2106.00219/) (2021)
- [Keep It Private: Unsupervised Privatization of Online Text](https://mlsanity.com/paper/2405.10260/) (2024)
- [Leveraging Long-Context Large Language Models for Multi-Document Understanding and Summarization in Enterprise Applications](https://mlsanity.com/paper/2409.18454/) (2024)
- [Information-Theoretic Distillation for Reference-less Summarization](https://mlsanity.com/paper/2403.13780/) (2024)
- [Dynamic Compressing Prompts for Efficient Inference of Large Language Models](https://mlsanity.com/paper/2504.11004/) (2025)
- [Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models](https://mlsanity.com/paper/2510.05090/) (2025)
- [Word-level Text Highlighting of Medical Texts for Telehealth Services](https://mlsanity.com/paper/2105.10400/) (2021)
- [AUGVIC: Exploiting BiText Vicinity for Low-Resource NMT](https://mlsanity.com/paper/2106.05141/) (2021)
- [TLDW: Extreme Multimodal Summarisation of News Videos](https://mlsanity.com/paper/2210.08481/) (2022)
- [Modeling Endorsement for Multi-Document Abstractive Summarization](https://mlsanity.com/paper/2110.07844/) (2021)
- [Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning](https://mlsanity.com/paper/2311.17514/) (2023)
- [Unsupervised Summarization Re-ranking](https://mlsanity.com/paper/2212.09593/) (2022)
- [Recent Trends in Unsupervised Summarization](https://mlsanity.com/paper/2305.11231/) (2023)
- [Efficient and Interpretable Compressive Text Summarisation with Unsupervised Dual-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.03415/) (2023)
- [JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences](https://mlsanity.com/paper/2508.16870/) (2025)

## Abstract (arXiv)

This work presents a new approach to unsupervised abstractive summarization based on maximizing a combination of coverage and fluency for a given length constraint. It introduces a novel method that encourages the inclusion of key terms from the original document into the summary: key terms are masked out of the original document and must be filled in by a coverage model using the current generated summary. A novel unsupervised training procedure leverages this coverage model along with a fluency model to generate and score summaries. When tested on popular news summarization datasets, the method outperforms previous unsupervised methods by more than 2 R-1 points, and approaches results of competitive supervised methods. Our model attains higher levels of abstraction with copied passages roughly two times shorter than prior work, and learns to compress and merge sentences without supervision.