mlsanityData to 14 Oct 2026

Toward Understanding In-context vs. In-weight Learning

arXiv 2410.23042 ↗First posted 30 Oct 2024cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

It has recently been demonstrated empirically that in-context learning emerges in transformers when certain distributional properties are present in the training data, but this ability can also diminish upon further training. We provide a new theoretical understanding of these phenomena by identifying simplified distributional properties that give rise to the emergence and eventual disappearance of in-context learning. We do so by first analyzing a simplified model that uses a gating mechanism to choose between an in-weight and an in-context predictor. Through a combination of a generalization error and regret analysis we identify conditions where in-context and in-weight learning emerge. These theoretical findings are then corroborated experimentally by comparing the behaviour of a full transformer on the simplified distributions to that of the stylized model, demonstrating aligned results. We then extend the study to a full large language model, showing how fine-tuning on various collections of natural language prompts can elicit similar in-context and in-weight learning behaviour.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

30 of 41 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022In-context Learning and Induction Headscited by 964
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2019Introduction to Online Convex Optimizationcited by 803
  7. 2022Transformers learn in-context by gradient descentcited by 732
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  10. 2023Larger language models do in-context learning differentlycited by 394
  11. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  12. 2023In-Context Learning Creates Task Vectorscited by 312
18 more
  1. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  2. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  3. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  4. 2024Many-Shot In-Context Learningcited by 206
  5. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  6. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 118
  7. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
  8. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  9. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 89
  10. 2023In-Context Learning through the Bayesian Prismcited by 83
  11. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
  12. 2015Gated Feedback Recurrent Neural Networkscited by 54
  13. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 50
  14. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  15. 2024Why Larger Language Models Do In-context Learning Differently?cited by 45
  16. 2024Dual Operating Modes of In-Context Learningcited by 43
  17. 2024One-layer transformers fail to solve the induction heads taskcited by 32
  18. 2023A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Taskscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 8.

19 papers.

Sort:

By area:

  1. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  2. 2025Strategy Coopetition Explains the Emergence and Transience of In-Context Learningcited by 17
  3. 2025General Intelligence Requires Reward-based Pretrainingcited by 12
  4. 2024Can foundation models actively gather information in interactive environments to test hypotheses?cited by 8
  5. 2025Understanding Prompt Tuning and In-Context Learning via Meta-Learningcited by 6
  6. 2026Generative AI collective behavior needs an interactionist paradigmcited by 6
  7. 2026Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformerscited by 5
  8. 2025Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worldscited by 4
  9. 2025Softmax ≥ Linear: Transformers may learn to classify in-context by kernel gradient descentcited by 4
  10. 2025AcademicEval: Live Long-Context LLM Benchmarkcited by 4
  11. 2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learningcited by 3
  12. 2025Context and Diversity Matter: The Emergence of In-Context Learning in World Modelscited by 3
7 more
  1. 2025Stability of In-Context Learning: A Spectral Coverage Perspectivecited by 2
  2. 2026To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoningcited by 1
  3. 2026Learning the ARTS of Search for Automated Discoverycited by 1
  4. 2026Reconciling In-Context and In-Weight Learning via Dual Representation Space Encodingcited by 0
  5. 2026Temporal Dependencies in In-Context Learning: The Role of Induction Headscited by 0
  6. 2026Quantifying Aleatoric Uncertainty of In-Context Learning for Robust Measure of LLM Prediction Confidencecited by 0
  7. 2026Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agentscited by 0