mlsanityData to 7 Oct 2026

One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention

arXiv 2307.03576 ↗First posted 7 Jul 2023cs.LGcited by 168

Not on the map. Why?

Abstract (arXiv)

Recent works have empirically analyzed in-context learning and shown that transformers trained on synthetic linear regression tasks can learn to implement ridge regression, which is the Bayes-optimal predictor, given sufficient capacity [Aky"urek et al., 2023], while one-layer transformers with linear self-attention and no MLP layer will learn to implement one step of gradient descent (GD) on a least-squares linear regression objective [von Oswald et al., 2022]. However, the theory behind these observations remains poorly understood. We theoretically study transformers with a single layer of linear self-attention, trained on synthetic noisy linear regression data. First, we mathematically show that when the covariates are drawn from a standard Gaussian distribution, the one-layer transformer which minimizes the pre-training loss will implement a single step of GD on the least-squares linear regression objective. Then, we find that changing the distribution of the covariates and weight vector to a non-isotropic Gaussian distribution has a strong impact on the learned algorithm: the global minimizer of the pre-training loss now implements a single step of textitpre-conditioned GD. However, if only the distribution of the responses is changed, then this does not have a large effect on the learned algorithm: even when the response comes from a more general family of textitnonlinear functions, the global minimizer of the pre-training loss still implements a single step of GD on a least-squares linear regression objective.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). Builds on 11 of its 16 references that are in the map. First posted on arXiv 7 Jul 2023.

Builds on

11 of 16 references.

References · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  4. 2022Transformers learn in-context by gradient descentcited by 714
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  7. 2023Larger language models do in-context learning differentlycited by 386
  8. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  9. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  10. 2022Transformers Learn Shortcuts to Automatacited by 260
  11. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 59; 2025: 59; 2026: 37.

168 papers.

Sort:

By area:

  1. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  2. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  3. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  4. 2023In-Context Convergence of Transformerscited by 112
  5. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  6. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  7. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  8. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  9. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  10. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  11. 2024Transformers Provably Solve Parity Efficiently with Chain of Thoughtcited by 49
  12. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
38 more
  1. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  2. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  3. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  4. 2024Dual Operating Modes of In-Context Learningcited by 42
  5. 2024Transformers are Universal In-context Learnerscited by 42
  6. 2024An Information-Theoretic Analysis of In-Context Learningcited by 39
  7. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 39
  8. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
  9. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  10. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 30
  11. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  12. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  13. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  14. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
  15. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 27
  16. 2025Taming Knowledge Conflicts in Language Modelscited by 27
  17. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  18. 2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
  19. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 23
  20. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  21. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 22
  22. 2024Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannotcited by 21
  23. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21
  24. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 20
  25. 2024Setting the Record Straight on Transformer Oversmoothingcited by 19
  26. 2024Looped ReLU MLPs May Be All You Need as Practical Programmable Computerscited by 19
  27. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  28. 2025In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learningcited by 17
  29. 2024How do Transformers perform In-Context Autoregressive Learning?cited by 16
  30. 2025Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thoughtcited by 16
  31. 2023Towards Understanding How Transformers Learn In-context Through a Representation Learning Lenscited by 15
  32. 2024On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixturescited by 15
  33. 2025Test-Time Training Provably Improves Transformers as In-context Learnerscited by 15
  34. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  35. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 14
  36. 2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 14
  37. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 13
  38. 2024Non-asymptotic Convergence of Training Transformers for Next-token Predictioncited by 13

50 most cited of 168 All 168 (JSON)