mlsanityData to 7 Oct 2026

In-context Learning and Induction Heads

arXiv 2209.11895 ↗First posted 24 Sep 2022cs.LGcited by 930

Not on the map. Why?

Cited by 100 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

"Induction heads" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all "in-context learning" in large transformer models (i.e. decreasing loss at increasing token indices). We find that induction heads develop at precisely the same point as a sudden sharp increase in in-context learning ability, visible as a bump in the training loss. We present six complementary lines of evidence, arguing that induction heads may be the mechanistic source of general in-context learning in transformer models of any size. For small attention-only models, we present strong, causal evidence; for larger models with MLPs, we present correlational evidence.

Key facts. Cited by 930 papers in the map (data through 7 Oct 2026). Builds on 8 of its 13 references that are in the map. First posted on arXiv 24 Sep 2022.

Builds on

8 of 13 references.

References · 8

  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  5. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  6. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  7. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  8. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 149; 2024: 224; 2025: 242; 2026: 299.

930 papers.

Sort:

By area:

  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  7. 2023StarCoder: may the source be with you!cited by 790
  8. 2023Large Language Modelscited by 774
  9. 2022A Survey on In-context Learningcited by 717
  10. 2022Transformers learn in-context by gradient descentcited by 714
  11. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
38 more
  1. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  2. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  3. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  4. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  5. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  6. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  7. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  8. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  9. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  10. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  11. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  12. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  13. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  14. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  15. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  16. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  17. 2024Massive Activations in Large Language Modelscited by 277
  18. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  19. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  20. 2023Challenges and Applications of Large Language Modelscited by 251
  21. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  22. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  23. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  24. 2023An Overview of Catastrophic AI Riskscited by 214
  25. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  26. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  27. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  28. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  29. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  30. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  31. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  32. 2022Talking About Large Language Modelscited by 174
  33. 2024The pitfalls of next-token predictioncited by 156
  34. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  35. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  36. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  37. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  38. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145

50 most cited of 930 All 930 (JSON)