mlsanityData to 14 Oct 2026

Exact Learning Dynamics of In-Context Learning in Linear Transformers and Its Application to Non-Linear Transformers

arXiv 2504.12916 ↗First posted 17 Apr 2025cs.LG cond-mat.dis-nncited by 3

Not on the map. Why?

Abstract (arXiv)

Transformer models exhibit remarkable in-context learning (ICL), adapting to novel tasks from examples within their context, yet the underlying mechanisms remain largely mysterious. Here, we provide an exact analytical characterization of ICL emergence by deriving the closed-form stochastic gradient descent (SGD) dynamics for a simplified linear transformer performing regression tasks. Our analysis reveals key properties: (1) a natural separation of timescales directly governed by the input data's covariance structure, leading to staged learning; (2) an exact description of how ICL develops, including fixed points corresponding to learned algorithms and conservation laws constraining the dynamics; and (3) surprisingly nonlinear learning behavior despite the model's linearity. We hypothesize this phenomenology extends to non-linear models. To test this, we introduce theory-inspired macroscopic measures (spectral rank dynamics, subspace stability) and use them to provide mechanistic explanations for (1) the sudden emergence of ICL in attention-only networks and (2) delayed generalization (grokking) in modular arithmetic models. Our work offers an exact dynamical model for ICL and theoretically grounded tools for analyzing complex transformer training.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

21 of 26 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2018On the Spectral Bias of Neural Networkscited by 1,341
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  9. 2022Transformers learn in-context by gradient descentcited by 732
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  12. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
9 more
  1. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  2. 2020Hopfield Networks is All You Needcited by 475
  3. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  4. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  5. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  6. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 74
  7. 2024Asymptotic theory of in-context learning by linear attentioncited by 55
  8. 2023Transformers learn through gradual rank increasecited by 52
  9. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamicscited by 2
  2. 2025Decomposing Prediction Mechanisms for In-Context Recallcited by 0
  3. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0