mlsanityData to 14 Oct 2026

Repetition Neurons: How Do Language Models Produce Repetitions?

arXiv 2410.13497 ↗First posted 17 Oct 2024cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

This paper introduces repetition neurons, regarded as skill neurons responsible for the repetition problem in text generation tasks. These neurons are progressively activated more strongly as repetition continues, indicating that they perceive repetition as a task to copy the previous context repeatedly, similar to in-context learning. We identify these repetition neurons by comparing activation values before and after the onset of repetition in texts generated by recent pre-trained language models. We analyze the repetition neurons in three English and one Japanese pre-trained language models and observe similar patterns across them.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

26 of 31 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  7. 2023In-Context Learning Creates Task Vectorscited by 312
  8. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  9. 2017Learning to Generate Reviews and Discovering Sentimentcited by 148
  10. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  11. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  12. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 108
14 more
  1. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 94
  2. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  3. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  4. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  5. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  6. 2024Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learningcited by 46
  7. 2023Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspectivecited by 39
  8. 2024LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMscited by 33
  9. 2023Understanding In-Context Learning from Repetitionscited by 29
  10. 2024Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editingcited by 18
  11. 2024From Loops to Oops: Fallback Behaviors of Language Models Under Uncertaintycited by 14
  12. 2023Humans and language models diverge when predicting repeating textcited by 13
  13. 2023Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generationcited by 13
  14. 2020Neural Text Generation with Artificial Negative Examplescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 11.

23 papers.

Sort:

By area:

  1. 2025Understanding the Repeat Curse in Large Language Models from a Feature Perspectivecited by 37
  2. 2025IPO: Your Language Model is Secretly a Preference Classifiercited by 17
  3. 2025Repetitions are not all alike: distinct mechanisms sustain repetition in language modelscited by 10
  4. 2025SPIRIT: Patching Speech Language Models against Jailbreak Attackscited by 9
  5. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMscited by 8
  6. 2026ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillationcited by 8
  7. 2025Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Modelscited by 7
  8. 2025What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformerscited by 7
  9. 2025CHD: Coupled Hierarchical Diffusion for Long-Horizon Taskscited by 6
  10. 2025Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learningcited by 5
  11. 2026From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Modelscited by 4
  12. 2025Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Modelscited by 3
11 more
  1. 2026Resource Consumption Threats in Large Language Modelscited by 2
  2. 2026LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Interventioncited by 2
  3. 2026Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Modelscited by 1
  4. 2026Temperature-driven inversion and nonlinear dynamics in ChatGPT-like AIscited by 1
  5. 2025Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errorscited by 0
  6. 2025In-Context Learning Without Copyingcited by 0
  7. 2026Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Modelscited by 0
  8. 2026Doc2FRC: Length-Consistent Document-Level Machine Translation via Fixed-Range Chunkingcited by 0
  9. 2026Target-Language Generation in Multilingual Models: Activation Steering and Optimal Controlcited by 0
  10. 2026Uncovering Uncontrolled Repetition through Residual Stream Dynamicscited by 0
  11. 2026Not All Answers Are Contextually Persuadable: Inference Dynamics in Large Language Models under Contextual Influencecited by 0