mlsanityData to 7 Oct 2026

Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning

arXiv 2507.07810 ↗First posted 10 Jul 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper investigates the relationship between large language models' (LLMs) ability to recognize repetitive input patterns and their performance on in-context learning (ICL). In contrast to prior work that has primarily focused on attention heads, we examine this relationship from the perspective of skill neurons, specifically repetition neurons. Our experiments reveal that the impact of these neurons on ICL performance varies depending on the depth of the layer in which they reside. By comparing the effects of repetition neurons and induction heads, we further identify strategies for reducing repetitive outputs while maintaining strong ICL capabilities.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

18 of 26 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2022In-context Learning and Induction Headscited by 930
  9. 2022A Survey on In-context Learningcited by 717
  10. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  11. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 90
  12. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
6 more
  1. 2024Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learningcited by 46
  2. 2023Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspectivecited by 39
  3. 2023Understanding In-Context Learning from Repetitionscited by 27
  4. 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 22
  5. 2021Relating Neural Text Degeneration to Exposure Biascited by 19
  6. 2024Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editingcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Resource Consumption Threats in Large Language Modelscited by 2
  2. 2026Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Failcited by 2
  3. 2026Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Modelscited by 1
  4. 2026When Can One Neuron Fix Repetition Loops in LLMs?cited by 0
  5. 2026Target-Language Generation in Multilingual Models: Activation Steering and Optimal Controlcited by 0