# Repetition Neurons: How Do Language Models Produce Repetitions?

arXiv: https://arxiv.org/abs/2410.13497 · HTML: https://mlsanity.com/paper/2410.13497/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Learning to Generate Reviews and Discovering Sentiment](https://mlsanity.com/paper/1704.01444/) (2017)
- [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022)
- [The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism](https://mlsanity.com/paper/2407.10457/) (2024)
- [Circuit Component Reuse Across Tasks in Transformer Language Models](https://mlsanity.com/paper/2310.08744/) (2023)
- [Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](https://mlsanity.com/paper/2206.02369/) (2022)
- [Neurons in Large Language Models: Dead, N-gram, Positional](https://mlsanity.com/paper/2309.04827/) (2023)
- [A Theoretical Analysis of the Repetition Problem in Text Generation](https://mlsanity.com/paper/2012.14660/) (2020)
- [Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale](https://mlsanity.com/paper/2212.09095/) (2022)
- [Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons](https://mlsanity.com/paper/2308.13198/) (2023)
- [Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning](https://mlsanity.com/paper/2407.07011/) (2024)
- [Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective](https://mlsanity.com/paper/2310.10226/) (2023)
- [LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs](https://mlsanity.com/paper/2407.03963/) (2024)
- [Understanding In-Context Learning from Repetitions](https://mlsanity.com/paper/2310.00297/) (2023)
- [Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing](https://mlsanity.com/paper/2410.07054/) (2024)
- [From Loops to Oops: Fallback Behaviors of Language Models Under Uncertainty](https://mlsanity.com/paper/2407.06071/) (2024)
- [Humans and language models diverge when predicting repeating text](https://mlsanity.com/paper/2310.06408/) (2023)
- [Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generation](https://mlsanity.com/paper/2310.14971/) (2023)
- [Neural Text Generation with Artificial Negative Examples](https://mlsanity.com/paper/2012.14124/) (2020)

## Built on by (most cited first, 23 of 23)

- [Understanding the Repeat Curse in Large Language Models from a Feature Perspective](https://mlsanity.com/paper/2504.14218/) (2025)
- [IPO: Your Language Model is Secretly a Preference Classifier](https://mlsanity.com/paper/2502.16182/) (2025)
- [Repetitions are not all alike: distinct mechanisms sustain repetition in language models](https://mlsanity.com/paper/2504.01100/) (2025)
- [SPIRIT: Patching Speech Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2505.13541/) (2025)
- [IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs](https://mlsanity.com/paper/2511.04727/) (2025)
- [ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation](https://mlsanity.com/paper/2607.13124/) (2026)
- [Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Models](https://mlsanity.com/paper/2503.20576/) (2025)
- [What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformers](https://mlsanity.com/paper/2506.13688/) (2025)
- [CHD: Coupled Hierarchical Diffusion for Long-Horizon Tasks](https://mlsanity.com/paper/2505.07261/) (2025)
- [Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning](https://mlsanity.com/paper/2507.07810/) (2025)
- [From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models](https://mlsanity.com/paper/2601.11020/) (2026)
- [Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models](https://mlsanity.com/paper/2505.13514/) (2025)
- [Resource Consumption Threats in Large Language Models](https://mlsanity.com/paper/2603.16068/) (2026)
- [LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Intervention](https://mlsanity.com/paper/2604.10044/) (2026)
- [Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models](https://mlsanity.com/paper/2602.23179/) (2026)
- [Temperature-driven inversion and nonlinear dynamics in ChatGPT-like AIs](https://mlsanity.com/paper/2608.00939/) (2026)
- [Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errors](https://mlsanity.com/paper/2502.19669/) (2025)
- [In-Context Learning Without Copying](https://mlsanity.com/paper/2511.05743/) (2025)
- [Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Models](https://mlsanity.com/paper/2601.17585/) (2026)
- [Doc2FRC: Length-Consistent Document-Level Machine Translation via Fixed-Range Chunking](https://mlsanity.com/paper/2609.12674/) (2026)
- [Target-Language Generation in Multilingual Models: Activation Steering and Optimal Control](https://mlsanity.com/paper/2609.16967/) (2026)
- [Uncovering Uncontrolled Repetition through Residual Stream Dynamics](https://mlsanity.com/paper/2609.38802/) (2026)
- [Not All Answers Are Contextually Persuadable: Inference Dynamics in Large Language Models under Contextual Influence](https://mlsanity.com/paper/2610.04791/) (2026)

## Abstract (arXiv)

This paper introduces repetition neurons, regarded as skill neurons responsible for the repetition problem in text generation tasks. These neurons are progressively activated more strongly as repetition continues, indicating that they perceive repetition as a task to copy the previous context repeatedly, similar to in-context learning. We identify these repetition neurons by comparing activation values before and after the onset of repetition in texts generated by recent pre-trained language models. We analyze the repetition neurons in three English and one Japanese pre-trained language models and observe similar patterns across them.