Distinct mechanisms underlying in-context learning in transformers
Not on the map. Why?
Abstract (arXiv)
Modern distributed networks, notably transformers, acquire a remarkable ability (termed `in-context learning') to adapt their computation to input statistics, such that a fixed network can be applied to data from a broad range of systems. Here, we provide a complete mechanistic characterization of this behavior in transformers trained on a finite set S of discrete Markov chains. The transformer displays four algorithmic phases, characterized by whether the network memorizes and generalizes, and whether it uses 1-point or 2-point statistics. We show that the four phases are implemented by multi-layer subcircuits that exemplify two qualitatively distinct mechanisms for implementing context-adaptive computations. Minimal models isolate the key features of both motifs. Memorization and generalization phases are delineated by two boundaries that depend on data diversity, K = |S|. The first (K1^ast) is set by a kinetic competition between subcircuits and the second (K2^ast) is set by a representational bottleneck. A symmetry-constrained theory of a transformer's training dynamics explains the sharp transition from 1-point to 2-point generalization and identifies key features of the loss landscape that allow the network to generalize. Put together, we show that transformers develop distinct subcircuits to implement in-context learning and identify conditions that favor certain mechanisms over others.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 14 Apr 2026.
Builds on
44 of 56 references.
References · 44
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2022Transformers learn in-context by gradient descentcited by 714
- 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
- 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
- 2023Trained Transformers Learn Linear Models In-Contextcited by 331
- 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
- 2023In-Context Learning Creates Task Vectorscited by 302
- 2023Function Vectors in Large Language Modelscited by 292
32 more
- 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
- 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
- 2023Localizing Model Behavior with Path Patchingcited by 182
- 2023Birth of a Transformer: A Memory Viewpointcited by 154
- 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
- 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
- 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
- 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
- 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
- 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
- 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
- 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
- 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
- 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
- 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 86
- 2025Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Trainingcited by 78
- 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
- 2019Meta-learning of Sequential Strategiescited by 68
- 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
- 2024Asymptotic theory of in-context learning by linear attentioncited by 54
- 2019Improving Transformer Models by Reordering their Sublayerscited by 52
- 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 52
- 2024Dual Operating Modes of In-Context Learningcited by 42
- 2024Zero-shot forecasting of chaotic systemscited by 25
- 2025Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Modelscited by 24
- 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
- 2024Differential learning kinetics govern the transition from memorization to generalization during in-context learningcited by 19
- 2025In-Context Learning Strategies Emerge Rationallycited by 15
- 2026Training Language Models via Neural Cellular Automatacited by 15
- 2025When can in-context learning generalize out of task distribution?cited by 13
- 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformercited by 12
- 2026Transformers for dynamical systems learn transfer operators in-contextcited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.