# Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning

arXiv: https://arxiv.org/abs/2507.07810 · HTML: https://mlsanity.com/paper/2507.07810/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Birth of a Transformer: A Memory Viewpoint](https://mlsanity.com/paper/2306.00802/) (2023)
- [Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](https://mlsanity.com/paper/2206.02369/) (2022)
- [Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale](https://mlsanity.com/paper/2212.09095/) (2022)
- [Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning](https://mlsanity.com/paper/2407.07011/) (2024)
- [Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective](https://mlsanity.com/paper/2310.10226/) (2023)
- [Understanding In-Context Learning from Repetitions](https://mlsanity.com/paper/2310.00297/) (2023)
- [Repetition Neurons: How Do Language Models Produce Repetitions?](https://mlsanity.com/paper/2410.13497/) (2024)
- [Relating Neural Text Degeneration to Exposure Bias](https://mlsanity.com/paper/2109.08705/) (2021)
- [Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing](https://mlsanity.com/paper/2410.07054/) (2024)

## Built on by (most cited first, 5 of 5)

- [Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail](https://mlsanity.com/paper/2608.23651/) (2026)
- [Resource Consumption Threats in Large Language Models](https://mlsanity.com/paper/2603.16068/) (2026)
- [Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models](https://mlsanity.com/paper/2602.23179/) (2026)
- [When Can One Neuron Fix Repetition Loops in LLMs?](https://mlsanity.com/paper/2606.13705/) (2026)
- [Target-Language Generation in Multilingual Models: Activation Steering and Optimal Control](https://mlsanity.com/paper/2609.16967/) (2026)

## Abstract (arXiv)

This paper investigates the relationship between large language models' (LLMs) ability to recognize repetitive input patterns and their performance on in-context learning (ICL). In contrast to prior work that has primarily focused on attention heads, we examine this relationship from the perspective of skill neurons, specifically repetition neurons. Our experiments reveal that the impact of these neurons on ICL performance varies depending on the depth of the layer in which they reside. By comparing the effects of repetition neurons and induction heads, we further identify strategies for reducing repetitive outputs while maintaining strong ICL capabilities.