# Self-Repetition in Abstractive Neural Summarizers

arXiv: https://arxiv.org/abs/2210.08145 · HTML: https://mlsanity.com/paper/2210.08145/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Builds on (most cited first, 15 of 15)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [A Theoretical Analysis of the Repetition Problem in Text Generation](https://mlsanity.com/paper/2012.14660/) (2020)
- [Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization](https://mlsanity.com/paper/2008.11293/) (2020)
- [A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization](https://mlsanity.com/paper/1707.07062/) (2017)
- [Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution](https://mlsanity.com/paper/2106.01518/) (2021)

## Built on by (most cited first, 13 of 13)

- [Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores](https://mlsanity.com/paper/2403.00553/) (2024)
- [Detection and Measurement of Syntactic Templates in Generated Text](https://mlsanity.com/paper/2407.00211/) (2024)
- [Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations](https://mlsanity.com/paper/2305.13693/) (2023)
- [Scaling Parameter-Constrained Language Models with Quality Data](https://mlsanity.com/paper/2410.03083/) (2024)
- [A Strategic Coordination Framework of Small LLMs Matches Large LLMs in Data Synthesis](https://mlsanity.com/paper/2504.12322/) (2025)
- [Improving Language Model Personas via Rationalization with Psychological Scaffolds](https://mlsanity.com/paper/2504.17993/) (2025)
- [Domain Regeneration: How well do LLMs match syntactic properties of text domains?](https://mlsanity.com/paper/2505.07784/) (2025)
- [Measuring AI "Slop" in Text](https://mlsanity.com/paper/2509.19163/) (2025)
- [Exploring Language Patterns of Prompts in Text-to-Image Generation and Their Impact on Visual Diversity](https://mlsanity.com/paper/2504.14125/) (2025)
- [Who Wrote the Book? Detecting and Attributing LLM Ghostwriters](https://mlsanity.com/paper/2603.28054/) (2026)
- [Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries](https://mlsanity.com/paper/2311.09458/) (2023)
- [SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection](https://mlsanity.com/paper/2603.22213/) (2026)
- [MultiGhostBench: A Multilingual Benchmark for Long-Form LLM-Generated Text Attribution under Distribution Shifts](https://mlsanity.com/paper/2609.02379/) (2026)

## Abstract (arXiv)

We provide a quantitative and qualitative analysis of self-repetition in the output of neural summarizers. We measure self-repetition as the number of n-grams of length four or longer that appear in multiple outputs of the same system. We analyze the behavior of three popular architectures (BART, T5, and Pegasus), fine-tuned on five datasets. In a regression analysis, we find that the three architectures have different propensities for repeating content across output summaries for inputs, with BART being particularly prone to self-repetition. Fine-tuning on more abstractive data, and on data featuring formulaic language, is associated with a higher rate of self-repetition. In qualitative analysis we find systems produce artefacts such as ads and disclaimers unrelated to the content being summarized, as well as formulaic phrases common in the fine-tuning domain. Our approach to corpus-level analysis of self-repetition may help practitioners clean up training data for summarizers and ultimately support methods for minimizing the amount of self-repetition.