mlsanityData to 14 Oct 2026

Transformers learn to implement preconditioned gradient descent for in-context learning

arXiv 2306.00297 ↗First posted 1 Jun 2023cs.LG cs.AIcited by 279

Not on the map. Why?

Abstract (arXiv)

Several recent works demonstrate that transformers can implement algorithms like gradient descent. By a careful construction of weights, these works show that multiple layers of transformers are expressive enough to simulate iterations of gradient descent. Going beyond the question of expressivity, we ask: Can transformers learn to implement such algorithms by training over random problem instances? To our knowledge, we make the first theoretical progress on this question via an analysis of the loss landscape for linear transformers trained over random instances of linear regression. For a single attention layer, we prove the global minimum of the training objective implements a single iteration of preconditioned gradient descent. Notably, the preconditioning matrix not only adapts to the input distribution but also to the variance induced by data inadequacy. For a transformer with L attention layers, we prove certain critical points of the training objective implement L iterations of preconditioned gradient descent. Our results call for future theoretical studies on learning algorithms by training transformers.

Key facts. Cited by 279 papers in the map (data through 14 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

19 of 34 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  5. 2022In-context Learning and Induction Headscited by 964
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  7. 2022Transformers learn in-context by gradient descentcited by 732
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  9. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  11. 2014Neural Turing Machinescited by 644
  12. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
7 more
  1. 2021MetaICL: Learning to Learn In Contextcited by 454
  2. 2023Looped Transformers as Programmable Computerscited by 237
  3. 2016Learning to Optimizecited by 195
  4. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  5. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  6. 2017Residual Connections Encourage Iterative Inferencecited by 77
  7. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 27; 2024: 88; 2025: 93; 2026: 71.

279 papers.

Sort:

By area:

  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  2. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 224
  3. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  4. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  5. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  6. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
  7. 2023In-Context Convergence of Transformerscited by 115
  8. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  9. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  10. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  11. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  12. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
38 more
  1. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  2. 2024Can large language models explore in-context?cited by 76
  3. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
  4. 2023Transformers as Support Vector Machinescited by 74
  5. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  6. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 65
  7. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  8. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 57
  9. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  10. 2024Asymptotic theory of in-context learning by linear attentioncited by 55
  11. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 49
  12. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 46
  13. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  14. 2024Is Mamba Capable of In-Context Learning?cited by 45
  15. 2024Why Larger Language Models Do In-context Learning Differently?cited by 45
  16. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  17. 2024Dual Operating Modes of In-Context Learningcited by 43
  18. 2024Transformers are Universal In-context Learnerscited by 43
  19. 2025Learning without training: The implicit dynamics of in-context learningcited by 43
  20. 2024An Information-Theoretic Analysis of In-Context Learningcited by 40
  21. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 37
  22. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  23. 2024In-Context Fine-Tuning for Time-Series Foundation Modelscited by 35
  24. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 33
  25. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  26. 2025Taming Knowledge Conflicts in Language Modelscited by 30
  27. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 30
  28. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 29
  29. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  30. 20244Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusioncited by 27
  31. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 25
  32. 2023When can transformers reason with abstract symbols?cited by 24
  33. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 24
  34. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  35. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 22
  36. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 22
  37. 2024Setting the Record Straight on Transformer Oversmoothingcited by 21
  38. 2024Differential learning kinetics govern the transition from memorization to generalization during in-context learningcited by 21

50 most cited of 279 All 279 (JSON)