mlsanityData to 14 Oct 2026

What learning algorithm is in-context learning? Investigations with linear models

arXiv 2211.15661 ↗First posted 28 Nov 2022cs.LG cs.CLcited by 684

Not on the map. Why?

Cited by 52 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural sequence models, especially transformers, exhibit a remarkable capacity for in-context learning. They can construct new predictors from sequences of labeled examples (x, f(x)) presented in the input without further parameter updates. We investigate the hypothesis that transformer-based in-context learners implement standard learning algorithms implicitly, by encoding smaller models in their activations, and updating these implicit models as new examples appear in the context. Using linear regression as a prototypical problem, we offer three sources of evidence for this hypothesis. First, we prove by construction that transformers can implement learning algorithms for linear models based on gradient descent and closed-form ridge regression. Second, we show that trained in-context learners closely match the predictors computed by gradient descent, ridge regression, and exact least-squares regression, transitioning between different predictors as transformer depth and dataset noise vary, and converging to Bayesian estimators for large widths and depths. Third, we present preliminary evidence that in-context learners share algorithmic features with these predictors: learners' late layers non-linearly encode weight vectors and moment matrices. These results suggest that in-context learning is understandable in algorithmic terms, and that (at least in the linear case) learners may rediscover standard estimation algorithms. Code and reference implementations are released at https://github.com/ekinakyurek/google-research/blob/master/incontext.

Key facts. Cited by 684 papers in the map (data through 14 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

24 of 34 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2016Layer Normalizationcited by 5,669
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
12 more
  1. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  2. 2016Learning to learn by gradient descent by gradient descentcited by 620
  3. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  4. 2021MetaICL: Learning to Learn In Contextcited by 454
  5. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  6. 2021Transformers Can Do Bayesian Inferencecited by 360
  7. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  8. 2020Compositional Explanations of Neuronscited by 172
  9. 2021Meta-learning via Language Model In-context Tuningcited by 164
  10. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  11. 2020Meta Learning Backpropagation And Improving Itcited by 49
  12. 2020Mask-Align: Self-Supervised Neural Word Alignmentcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 150; 2024: 197; 2025: 170; 2026: 161.

684 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023Large Language Modelscited by 780
  3. 2022Transformers learn in-context by gradient descentcited by 732
  4. 2022A Survey on In-context Learningcited by 721
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  6. 2023Larger language models do in-context learning differentlycited by 394
  7. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  8. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  9. 2023In-Context Learning Creates Task Vectorscited by 312
  10. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 309
  11. 2023Function Vectors in Large Language Modelscited by 308
  12. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
38 more
  1. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  2. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  3. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  4. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  5. 2023Looped Transformers as Programmable Computerscited by 237
  6. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  7. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 224
  8. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  9. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  10. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 207
  11. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  12. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  13. 2023Large Language Models as General Pattern Machinescited by 197
  14. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  15. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  16. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  17. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  18. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  19. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  20. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  21. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  22. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  23. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  24. 2023The Learnability of In-Context Learningcited by 127
  25. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  26. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  27. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 118
  28. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
  29. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  30. 2023In-Context Convergence of Transformerscited by 115
  31. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  32. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  33. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 104
  34. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 102
  35. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  36. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  37. 2025DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Processcited by 100
  38. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98

50 most cited of 684 All 684 (JSON)