mlsanityData to 7 Oct 2026

Dynamics of Transient Structure in In-Context Linear Regression Transformers

arXiv 2501.17745 ↗First posted 29 Jan 2025cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Modern deep neural networks display striking examples of rich internal computational structure. Uncovering principles governing the development of such structure is a priority for the science of deep learning. In this paper, we explore the transient ridge phenomenon: when transformers are trained on in-context linear regression tasks with intermediate task diversity, they initially behave like ridge regression before specializing to the tasks in their training distribution. This transition from a general solution to a specialized solution is revealed by joint trajectory principal component analysis. Further, we draw on the theory of Bayesian internal model selection to suggest a general explanation for the phenomena of transient structure in transformers, based on an evolving tradeoff between loss and complexity. We empirically validate this explanation by measuring the model complexity of our transformers as defined by the local learning coefficient.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 31 of its 62 references that are in the map. First posted on arXiv 29 Jan 2025.

Builds on

31 of 62 references.

References · 31

  1. 2022In-context Learning and Induction Headscited by 930
  2. 2022Toy Models of Superpositioncited by 928
  3. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  4. 2022Transformers learn in-context by gradient descentcited by 714
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  8. 2017Exploring Generalization in Deep Learningcited by 595
  9. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  10. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  11. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  12. 2018A mathematical theory of semantic development in deep neural networkscited by 182
19 more
  1. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  2. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  3. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  4. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  5. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  6. 2023In-Context Learning through the Bayesian Prismcited by 78
  7. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
  8. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  9. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 52
  10. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  11. 2020Deep Learning is Singular, and That's Goodcited by 49
  12. 2024Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscited by 40
  13. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 39
  14. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 39
  15. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 28
  16. 2023The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifoldcited by 20
  17. 2024Differential learning kinetics govern the transition from memorization to generalization during in-context learningcited by 19
  18. 2018Revisiting the Gelman-Rubin Diagnosticcited by 18
  19. 2018PCA of high dimensional random walks with comparison to neural network trainingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025The Loss Kernel: A Geometric Probe for Deep Learning Interpretabilitycited by 8
  2. 2025From Global to Local: A Scalable Benchmark for Local Posterior Samplingcited by 5
  3. 2025Influence Dynamics and Stagewise Data Attributioncited by 5
  4. 2025Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theorycited by 4
  5. 2026Stagewise Reinforcement Learning and the Geometry of the Regret Landscapecited by 3
  6. 2025Modes of Sequence Models and Learning Coefficientscited by 2
  7. 2025Softmax ≥ Linear: Transformers may learn to classify in-context by kernel gradient descentcited by 2
  8. 2026Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledgecited by 2
  9. 2026Mechanistic Anomaly Detection via Functional Attributioncited by 1
  10. 2026What does a Bayes-filtered transformer believe? A predictive Monte Carlo approachcited by 1
  11. 2026Temporal Task Diversity: Inductive Biases Under Non-Stationarity in Synthetic Sequence Modellingcited by 0
  12. 2026Structure and Scale in Simplicial Sequence Modellingcited by 0