mlsanityData to 7 Oct 2026

Trained Transformers Learn Linear Models In-Context

arXiv 2306.09927 ↗First posted 16 Jun 2023stat.ML cs.AI cs.CLcited by 331

Not on the map. Why?

Cited by 23 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attention-based neural networks such as transformers have demonstrated a remarkable ability to exhibit in-context learning (ICL): Given a short prompt sequence of tokens from an unseen task, they can formulate relevant per-token and next-token predictions without any parameter updates. By embedding a sequence of labeled training data and unlabeled test data as a prompt, this allows for transformers to behave like supervised learning algorithms. Indeed, recent work has shown that when training transformer architectures over random instances of linear regression problems, these models' predictions mimic those of ordinary least squares. Towards understanding the mechanisms underlying this phenomenon, we investigate the dynamics of ICL in transformers with a single linear self-attention layer trained by gradient flow on linear regression tasks. We show that despite non-convexity, gradient flow with a suitable random initialization finds a global minimum of the objective function. At this global minimum, when given a test prompt of labeled examples from a new prediction task, the transformer achieves prediction error competitive with the best linear predictor over the test prompt distribution. We additionally characterize the robustness of the trained transformer to a variety of distribution shifts and show that although a number of shifts are tolerated, shifts in the covariate distribution of the prompts are not. Motivated by this, we consider a generalized ICL setting where the covariate distributions can vary across prompts. We show that although gradient flow succeeds at finding a global minimum in this setting, the trained transformer is still brittle under mild covariate shifts. We complement this finding with experiments on large, nonlinear transformer architectures which we show are more robust under covariate shifts.

Key facts. Cited by 331 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

39 of 52 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2022Transformers learn in-context by gradient descentcited by 714
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  11. 2018Universal Transformerscited by 613
  12. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
27 more
  1. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  2. 2017Implicit Regularization in Matrix Factorizationcited by 318
  3. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  4. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  5. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  6. 2022Transformers Learn Shortcuts to Automatacited by 260
  7. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  8. 2022Exploring Length Generalization in Large Language Modelscited by 204
  9. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  10. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  11. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  12. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  13. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  14. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  15. 2022Vision Transformers provably learn spatial structurecited by 92
  16. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  17. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  18. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 82
  19. 2023In-Context Learning through the Bayesian Prismcited by 78
  20. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  21. 2023Mimetic Initialization of Self-Attention Layerscited by 52
  22. 2021On the Expressive Power of Self-Attention Matricescited by 44
  23. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  24. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  25. 2023A Closer Look at In-Context Learning under Distribution Shiftscited by 25
  26. 2023A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Taskscited by 17
  27. 2020On implicit regularization: Morse functions and applications to matrix factorizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 35; 2024: 112; 2025: 107; 2026: 77.

331 papers.

Sort:

By area:

  1. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  2. 2024Many-Shot In-Context Learningcited by 203
  3. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  4. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  5. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  6. 2023In-Context Convergence of Transformerscited by 112
  7. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  8. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  9. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  10. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  11. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  12. 2024Can large language models explore in-context?cited by 75
38 more
  1. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  2. 2023Transformers as Support Vector Machinescited by 67
  3. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  4. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  5. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  6. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  7. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  8. 2024Transformers Provably Solve Parity Efficiently with Chain of Thoughtcited by 49
  9. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  10. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  11. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  12. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  13. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  14. 2024Dual Operating Modes of In-Context Learningcited by 42
  15. 2024Transformers are Universal In-context Learnerscited by 42
  16. 2025Learning without training: The implicit dynamics of in-context learningcited by 41
  17. 2024An Information-Theoretic Analysis of In-Context Learningcited by 39
  18. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 39
  19. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 39
  20. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  21. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  22. 2023Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selectioncited by 36
  23. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  24. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
  25. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 36
  26. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  27. 2023How are Prompts Different in Terms of Sensitivity?cited by 34
  28. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Gamescited by 33
  29. 2024The broader spectrum of in-context learningcited by 33
  30. 2023In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trickcited by 32
  31. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  32. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  33. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  34. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  35. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 30
  36. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 29
  37. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  38. 2024Linear Transformers are Versatile In-Context Learnerscited by 28

50 most cited of 331 All 331 (JSON)