mlsanityData to 7 Oct 2026

Predicting the Emergence of Induction Heads in Language Model Pretraining

arXiv 2511.16893 ↗First posted 21 Nov 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise characterization of their emergence, especially in the context of language modeling, remains wanting. In this study, we investigate the relationship between statistical properties of the training data and IH formation in both natural and synthetic training data settings. We show that: (1) a simple equation combining batch size and context size predicts the point at which IHs form and that this emergence point is agnostic to model size; (2) surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find an effective decision boundary in terms of these two values; (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but categoriality and the shape of the marginal distribution appear to modulate IH formation near the decision boundary.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 31 of its 33 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

31 of 33 references.

References · 31

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  11. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  12. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
19 more
  1. 2018An Empirical Model of Large-Batch Trainingcited by 268
  2. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  3. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  4. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  5. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  6. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  7. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  8. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 86
  9. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  10. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 52
  11. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 34
  12. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  13. 2024One-layer transformers fail to solve the induction heads taskcited by 30
  14. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
  15. 2025Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Trainingcited by 20
  16. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  17. 2025Language Models Grow Less Humanlike beyond Phase Transitioncited by 10
  18. 2025Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Modelscited by 5
  19. 2025What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chainscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Developmental Trajectories of Situation Modeling and Mentalizing in Transformer Language Modelscited by 0
  2. 2026The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Modelscited by 0
  3. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 0
  4. 2026The Token Before the Value Is the Key: How Hybrid Architectures Organize Induction Circuitscited by 0
  5. 2026What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetitioncited by 0