mlsanityData to 14 Oct 2026

Strategy Coopetition Explains the Emergence and Transience of In-Context Learning

arXiv 2503.05631 ↗First posted 7 Mar 2025cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) is a powerful ability that emerges in transformer models, enabling them to learn from context without weight updates. Recent work has established emergent ICL as a transient phenomenon that can sometimes disappear after long training times. In this work, we sought a mechanistic understanding of these transient dynamics. Firstly, we find that, after the disappearance of ICL, the asymptotic strategy is a remarkable hybrid between in-weights and in-context learning, which we term "context-constrained in-weights learning" (CIWL). CIWL is in competition with ICL, and eventually replaces it as the dominant strategy of the model (thus leading to ICL transience). However, we also find that the two competing strategies actually share sub-circuits, which gives rise to cooperative dynamics as well. For example, in our setup, ICL is unable to emerge quickly on its own, and can only be enabled through the simultaneous slow development of asymptotic CIWL. CIWL thus both cooperates and competes with ICL, a phenomenon we term "strategy coopetition." We propose a minimal mathematical model that reproduces these key dynamics and interactions. Informed by this model, we were able to identify a setup where ICL is truly emergent and persistent.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

22 of 31 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2022Toy Models of Superpositioncited by 971
  7. 2022In-context Learning and Induction Headscited by 964
  8. 2023Larger language models do in-context learning differentlycited by 394
  9. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  10. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  11. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 89
  12. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
10 more
  1. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  2. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 54
  3. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  4. 2024Dual Operating Modes of In-Context Learningcited by 43
  5. 2024Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscited by 40
  6. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  7. 2024The broader spectrum of in-context learningcited by 35
  8. 2024Differential learning kinetics govern the transition from memorization to generalization during in-context learningcited by 21
  9. 2024Toward Understanding In-context vs. In-weight Learningcited by 19
  10. 2024Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgettingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025Just-in-time and distributed task representations in language modelscited by 9
  2. 2025Graph Diffusion Transformers are In-Context Molecular Designerscited by 7
  3. 2025Understanding Prompt Tuning and In-Context Learning via Meta-Learningcited by 6
  4. 2025Distinct Computations Emerge From Compositional Curricula in In-Context Learningcited by 5
  5. 2025Softmax ≥ Linear: Transformers may learn to classify in-context by kernel gradient descentcited by 4
  6. 2025Latent Concept Disentanglement in Transformer-based Language Modelscited by 3
  7. 2025Context and Diversity Matter: The Emergence of In-Context Learning in World Modelscited by 3
  8. 2026Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledgecited by 3
  9. 2026Invariant Learning Dynamics of Transformers in Inductive Reasoning Taskscited by 2
  10. 2025An evolutionary perspective on modes of learning in Transformerscited by 1
  11. 2025Multi-Layer Attention is the Amplifier of Demonstration Effectivenesscited by 1
  12. 2025In-Context Learning Without Copyingcited by 0
5 more
  1. 2026In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Taskscited by 0
  2. 2026Design-Specification Tiling for ICL-based CAD Code Generationcited by 0
  3. 2026Reconciling In-Context and In-Weight Learning via Dual Representation Space Encodingcited by 0
  4. 2026Contrast Matters: Understanding Robustness of In-Context Fine-Tuning to Target-Context Relatednesscited by 0
  5. 2026Pretraining Curricula Enable Selective Fine-tuningcited by 0