mlsanityData to 14 Oct 2026

Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning

arXiv 2407.07011 ↗First posted 9 Jul 2024cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown a remarkable ability to learn and perform complex tasks through in-context learning (ICL). However, a comprehensive understanding of its internal mechanisms is still lacking. This paper explores the role of induction heads in a few-shot ICL setting. We analyse two state-of-the-art models, Llama-3-8B and InternLM2-20B on abstract pattern recognition and NLP tasks. Our results show that even a minimal ablation of induction heads leads to ICL performance decreases of up to ~32% for abstract pattern recognition tasks, bringing the performance close to random. For NLP tasks, this ablation substantially decreases the model's ability to benefit from examples, bringing few-shot ICL performance close to that of zero-shot prompts. We further use attention knockout to disable specific induction patterns, and present fine-grained evidence for the role that the induction mechanism plays in ICL.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Jul 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 21; 2026: 15.

46 papers.

Sort:

By area:

  1. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 95
  2. 2024Attention Heads of Large Language Models: A Surveycited by 67
  3. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  4. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  5. 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 23
  6. 2024Looped ReLU MLPs May Be All You Need as Practical Programmable Computerscited by 19
  7. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  8. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformercited by 12
  9. 2025Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2Bcited by 10
  10. 2025Repetitions are not all alike: distinct mechanisms sustain repetition in language modelscited by 10
  11. 2025Rethinking Circuit Completeness in Language Models: AND, OR, and ADDER Gatescited by 10
  12. 2025Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergencecited by 10
34 more
  1. 2026Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Unitscited by 10
  2. 2025How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Headscited by 9
  3. 2024How Transformers Get Rich: Approximation and Dynamics Analysiscited by 8
  4. 2024KV Shifting Attention Enhances Language Modelingcited by 8
  5. 2025Understanding In-context Learning of Addition via Activation Subspacescited by 8
  6. 2025Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMscited by 6
  7. 2025Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilitiescited by 6
  8. 2025Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learningcited by 5
  9. 2026Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformerscited by 5
  10. 2026Universal Redundancies in Time Series Foundation Modelscited by 5
  11. 2025Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Trainingcited by 4
  12. 2025Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Modelscited by 3
  13. 2025The Initialization Determines Whether In-Context Learning Is Gradient Descentcited by 3
  14. 2026Do LLMs Know Tool Irrelevance? Demystifying Structural Alignment Bias in Tool Invocationscited by 3
  15. 2026Many-Shot CoT-ICL: Making In-Context Learning Truly Learncited by 3
  16. 2025Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learningcited by 2
  17. 2025Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysiscited by 2
  18. 2026Resource Consumption Threats in Large Language Modelscited by 2
  19. 2026Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalizationcited by 2
  20. 2024Skill Path: Unveiling Language Skills from Circuit Graphscited by 1
  21. 2024Universal Response and Emergence of Induction in LLMscited by 1
  22. 2026Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Modelscited by 1
  23. 2026In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Headscited by 1
  24. 2026Sentence-Level Contextual Entrainment in Large Language Modelscited by 1
  25. 2026Structural priors for data-efficient language learningcited by 1
  26. 2025Investigating Neurons and Heads in Transformer-based LLMs for Typographical Errorscited by 0
  27. 2025Geometric SSM: LTI State Space Models for Selective Taskscited by 0
  28. 2025Medical Interpretability and Knowledge Maps of Large Language Modelscited by 0
  29. 2025Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Modelscited by 0
  30. 2025In-Context Learning Without Copyingcited by 0
  31. 2026Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Inductioncited by 0
  32. 2026Temporal Dependencies in In-Context Learning: The Role of Induction Headscited by 0
  33. 2026Mirror, Mirror on the Wall: Prompt Echoing in Small Instruct Language Modelscited by 0
  34. 2026Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparitiescited by 0