# Repetitions are not all alike: distinct mechanisms sustain repetition in language models

arXiv: https://arxiv.org/abs/2504.01100 · HTML: https://mlsanity.com/paper/2504.01100/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 23 of 23)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism](https://mlsanity.com/paper/2407.10457/) (2024)
- [Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](https://mlsanity.com/paper/2206.02369/) (2022)
- [A Theoretical Analysis of the Repetition Problem in Text Generation](https://mlsanity.com/paper/2012.14660/) (2020)
- [The MiniPile Challenge for Data-Efficient Language Models](https://mlsanity.com/paper/2304.08442/) (2023)
- [Repetition Improves Language Model Embeddings](https://mlsanity.com/paper/2402.15449/) (2024)
- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [Which Attention Heads Matter for In-Context Learning?](https://mlsanity.com/paper/2502.14010/) (2025)
- [Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning](https://mlsanity.com/paper/2407.07011/) (2024)
- [Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals](https://mlsanity.com/paper/2402.11655/) (2024)
- [Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective](https://mlsanity.com/paper/2310.10226/) (2023)
- [Understanding In-Context Learning from Repetitions](https://mlsanity.com/paper/2310.00297/) (2023)
- [Repetition Neurons: How Do Language Models Produce Repetitions?](https://mlsanity.com/paper/2410.13497/) (2024)
- [Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing](https://mlsanity.com/paper/2410.07054/) (2024)
- [From Loops to Oops: Fallback Behaviors of Language Models Under Uncertainty](https://mlsanity.com/paper/2407.06071/) (2024)
- [Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generation](https://mlsanity.com/paper/2310.14971/) (2023)
- [Unnatural language processing: How do language models handle machine-generated prompts?](https://mlsanity.com/paper/2310.15829/) (2023)
- [Taming Repetition in Dialogue Generation](https://mlsanity.com/paper/2112.08657/) (2021)
- [Evil twins are not that evil: Qualitative insights into machine-generated prompts](https://mlsanity.com/paper/2412.08127/) (2024)

## Built on by (most cited first, 10 of 10)

- [Next Edit Prediction: Learning to Predict Code Edits from Context and Interaction History](https://mlsanity.com/paper/2508.10074/) (2025)
- [Word Salad Chopper: Reasoning Models Waste A Ton Of Decoding Budget On Useless Repetitions, Self-Knowingly](https://mlsanity.com/paper/2511.00536/) (2025)
- [Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery](https://mlsanity.com/paper/2606.06267/) (2026)
- [Resource Consumption Threats in Large Language Models](https://mlsanity.com/paper/2603.16068/) (2026)
- [SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs](https://mlsanity.com/paper/2604.07737/) (2026)
- [Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning](https://mlsanity.com/paper/2607.19345/) (2026)
- [Prompt-Model Interaction Reaches the Fixed Points: A deterministic, task-free structural readout -- and the factorizations of it that failed](https://mlsanity.com/paper/2608.21315/) (2026)
- [Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Models](https://mlsanity.com/paper/2601.17585/) (2026)
- [Improving LLM Code Reasoning via Semantic Equivalence Self-Play with Formal Verification](https://mlsanity.com/paper/2604.17010/) (2026)
- [Automatic or Controlled? Repetition Priming Reveals Divergent Processing in Base LLMs, Instruct LLMs, and Humans](https://mlsanity.com/paper/2608.14681/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) can sometimes degrade into repetitive loops, persistently generating identical word sequences. Because repetition is rare in natural human language, its frequent occurrence across diverse tasks and contexts in LLMs remains puzzling. Here we investigate whether behaviorally similar repetition patterns arise from distinct underlying mechanisms and how these mechanisms develop during model training. We contrast two conditions: repetitions elicited by natural text prompts with those induced by in-context learning (ICL) setups that explicitly require copying behavior. Our analyses reveal that ICL-induced repetition relies on a dedicated network of attention heads that progressively specialize over training, whereas naturally occurring repetition emerges early and lacks a defined circuitry. Attention inspection further shows that natural repetition focuses disproportionately on low-information tokens, suggesting a fallback behavior when relevant context cannot be retrieved. These results indicate that superficially similar repetition behaviors originate from qualitatively different internal processes, reflecting distinct modes of failure and adaptation in language models.