mlsanityData to 7 Oct 2026

Transformers learn in-context by gradient descent

arXiv 2212.07677 ↗First posted 15 Dec 2022cs.LG cs.AI cs.CLcited by 714

Not on the map. Why?

Cited by 36 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on auto-regressive objectives is closely related to gradient-based meta-learning formulations. We start by providing a simple weight construction that shows the equivalence of data transformations induced by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a regression loss. Motivated by that construction, we show empirically that when training self-attention-only Transformers on simple regression tasks either the models learned by GD and Transformers show great similarity or, remarkably, the weights found by optimization match the construction. Thus we show how trained Transformers become mesa-optimizers i.e. learn models by gradient descent in their forward pass. This allows us, at least in the domain of regression problems, to mechanistically understand the inner workings of in-context learning in optimized Transformers. Building on this insight, we furthermore identify how Transformers surpass the performance of plain gradient descent by learning an iterative curvature correction and learn linear models on deep data representations to solve non-linear regression tasks. Finally, we discuss intriguing parallels to a mechanism identified to be crucial for in-context learning termed induction-head (Olsson et al., 2022) and show how it could be understood as a specific case of in-context learning by gradient descent learning within Transformers. Code to reproduce the experiments can be found at https://github.com/google-research/self-organising-systems/tree/master/transformers_learn_icl_by_gd .

Key facts. Cited by 714 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

43 of 57 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2022In-context Learning and Induction Headscited by 930
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
31 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2016Learning to learn by gradient descent by gradient descentcited by 616
  3. 2019Deep Equilibrium Modelscited by 603
  4. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  5. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 588
  6. 2019Graph Transformer Networkscited by 520
  7. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  9. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  10. 2020Hopfield Networks is All You Needcited by 461
  11. 2018Meta-learning with differentiable closed-form solverscited by 395
  12. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  13. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  14. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  15. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  16. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  17. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  18. 2021Dive into Deep Learningcited by 111
  19. 2018Meta-Learning Probabilistic Inference For Predictioncited by 108
  20. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  21. 2019Meta-Learning with Warped Gradient Descentcited by 98
  22. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  23. 2019Deep Declarative Networks: A New Hopecited by 77
  24. 2019Meta-Curvaturecited by 70
  25. 2017Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithmcited by 65
  26. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  27. 2020Meta Learning Backpropagation And Improving Itcited by 49
  28. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  29. 2021Learning where to learn: Gradient sparsity in meta and continual learningcited by 33
  30. 2022Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagationcited by 26
  31. 2022Random initialisations performing above chance and how to find themcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 128; 2024: 195; 2025: 200; 2026: 189.

714 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022A Survey on In-context Learningcited by 717
  3. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  4. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  5. 2023Larger language models do in-context learning differentlycited by 386
  6. 2023Language Models can Solve Computer Taskscited by 371
  7. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  8. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  9. 2023Function Vectors in Large Language Modelscited by 292
  10. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  12. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
38 more
  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  3. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  4. 2023Looped Transformers as Programmable Computerscited by 224
  5. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  6. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  7. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  8. 2024Many-Shot In-Context Learningcited by 203
  9. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  10. 2023Large Language Models as General Pattern Machinescited by 196
  11. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  12. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  13. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  14. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  15. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  16. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  17. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  18. 2023The Learnability of In-Context Learningcited by 126
  19. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  20. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 124
  21. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  22. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  23. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  24. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  25. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  26. 2023In-Context Convergence of Transformerscited by 112
  27. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  28. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  29. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  30. 2023The Expressive Power of Low-Rank Adaptationcited by 100
  31. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  32. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  33. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  34. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  35. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  36. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  37. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  38. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82

50 most cited of 714 All 714 (JSON)