mlsanityData to 7 Oct 2026

The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains

arXiv 2402.11004 ↗First posted 16 Feb 2024cs.LGcited by 115

Not on the map. Why?

Abstract (arXiv)

Large language models have the ability to generate text that mimics patterns in their inputs. We introduce a simple Markov Chain sequence modeling task in order to study how this in-context learning (ICL) capability emerges. In our setting, each example is sampled from a Markov chain drawn from a prior distribution over Markov chains. Transformers trained on this task form statistical induction heads which compute accurate next-token probabilities given the bigram statistics of the context. During the course of training, models pass through multiple phases: after an initial stage in which predictions are uniform, they learn to sub-optimally predict using in-context single-token statistics (unigrams); then, there is a rapid phase transition to the correct in-context bigram solution. We conduct an empirical and theoretical investigation of this multi-phase process, showing how successful learning results from the interaction between the transformer's layers, and uncovering evidence that the presence of the simpler unigram solution may delay formation of the final bigram solution. We examine how learning is affected by varying the prior distribution over Markov chains, and consider the generalization of our in-context learning of Markov chains (ICL-MC) task to n-grams for n > 2.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 33 of its 47 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

33 of 47 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018Self-Attention with Relative Position Representationscited by 1,112
  5. 2017A Closer Look at Memorization in Deep Networkscited by 936
  6. 2022In-context Learning and Induction Headscited by 930
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2022Transformers learn in-context by gradient descentcited by 714
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  11. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  12. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
21 more
  1. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  2. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  3. 2023In-Context Learning Creates Task Vectorscited by 302
  4. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  5. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  6. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  7. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  8. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  9. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  10. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  11. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  12. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  13. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  14. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  15. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  16. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  17. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 78
  18. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  19. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  20. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 39
  21. 2023A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Taskscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 58; 2026: 27.

115 papers.

Sort:

By area:

  1. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  2. 2024Attention Heads of Large Language Models: A Surveycited by 65
  3. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  4. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 39
  5. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  6. 2024Large Language Models as Markov Chainscited by 33
  7. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  8. 2024Iteration Head: A Mechanistic Study of Chain-of-Thoughtcited by 31
  9. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  10. 2024Transformers on Markov Data: Constant Depth Sufficescited by 26
  11. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  12. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
38 more
  1. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  2. 2024Understanding Transformers via N-gram Statisticscited by 23
  3. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  4. 2024Toward Understanding In-context vs. In-weight Learningcited by 19
  5. 2024Differential learning kinetics govern the transition from memorization to generalization during in-context learningcited by 19
  6. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15
  7. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 15
  8. 2024Implicit Optimization Bias of Next-Token Prediction in Linear Modelscited by 14
  9. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 14
  10. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 14
  11. 2025HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinkingcited by 14
  12. 2025Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learningcited by 14
  13. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  14. 2025When can in-context learning generalize out of task distribution?cited by 13
  15. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  16. 2024Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layerscited by 12
  17. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 12
  18. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformercited by 12
  19. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 12
  20. 2024Can Transformers Learn n-gram Language Models?cited by 11
  21. 2025Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergencecited by 10
  22. 2025From Markov to Laplace: How Mamba In-Context Learns Markov Chainscited by 9
  23. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 9
  24. 2024Transformers Handle Endogeneity in In-Context Linear Regressioncited by 8
  25. 2025Rethinking Circuit Completeness in Language Models: AND, OR, and ADDER Gatescited by 8
  26. 2025Mechanistic evaluation of Transformers and state space modelscited by 8
  27. 2024How Transformers Get Rich: Approximation and Dynamics Analysiscited by 7
  28. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scalingcited by 7
  29. 2025Just-in-time and distributed task representations in language modelscited by 7
  30. 2025Selective Induction Heads: How Transformers Select Causal Structures In Contextcited by 7
  31. 2026To Use or not to Use Muon: How Simplicity Bias in Optimizers Matterscited by 7
  32. 2024N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needscited by 6
  33. 2025From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformerscited by 6
  34. 2024In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learningcited by 5
  35. 2025What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformerscited by 5
  36. 2025Position: We Need An Algorithmic Understanding of Generative AIcited by 5
  37. 2025Predicting the Emergence of Induction Heads in Language Model Pretrainingcited by 5
  38. 2026Large Vision-Language Models Get Lost in Attentioncited by 5

50 most cited of 115 All 115 (JSON)