mlsanityData to 14 Oct 2026

Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression

arXiv 2310.17086 ↗First posted 26 Oct 2023cs.LG cs.AI cs.CLcited by 49

On the map: Offline RL and imitation learning from data → Robot learning from demonstration

Abstract (arXiv)

Transformers excel at in-context learning (ICL) -- learning from demonstrations without parameter updates -- but how they do so remains a mystery. Recent work suggests that Transformers may internally run Gradient Descent (GD), a first-order optimization method, to perform ICL. In this paper, we instead demonstrate that Transformers learn to approximate second-order optimization methods for ICL. For in-context linear regression, Transformers share a similar convergence rate as Iterative Newton's Method, both exponentially faster than GD. Empirically, predictions from successive Transformer layers closely match different iterations of Newton's Method linearly, with each middle layer roughly computing 3 iterations; thus, Transformers and Newton's method converge at roughly the same rate. In contrast, Gradient Descent converges exponentially more slowly. We also show that Transformers can learn in-context on ill-conditioned data, a setting where Gradient Descent struggles but Iterative Newton succeeds. Finally, to corroborate our empirical findings, we prove that Transformers can implement k iterations of Newton's method with k + O(1) layers.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 62 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

43 of 62 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  12. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
31 more
  1. 2022Transformers learn in-context by gradient descentcited by 732
  2. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  4. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  5. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  6. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  8. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  9. 2021MetaICL: Learning to Learn In Contextcited by 454
  10. 2023Larger language models do in-context learning differentlycited by 394
  11. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  12. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  13. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  14. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  15. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  16. 2023Looped Transformers as Programmable Computerscited by 237
  17. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  18. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  19. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  20. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  21. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  22. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  23. 2023Transformers as Support Vector Machinescited by 74
  24. 2023In-context Example Selection with Influencescited by 61
  25. 2022Data Curation Alone Can Stabilize In-context Learningcited by 59
  26. 2023Max-Margin Token Selection in Attention Mechanismcited by 58
  27. 2023A Study on ReLU and Softmax in Transformercited by 53
  28. 2023Replacing softmax with ReLU in Vision Transformerscited by 44
  29. 2019Memory-Sample Tradeoffs for Linear Regression with Small Errorcited by 22
  30. 2022How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformerscited by 21
  31. 2023Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Abilitycited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 19; 2026: 16.

49 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  2. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  3. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
  4. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 65
  5. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  6. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 38
  7. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  8. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  9. 2024In-Context Symbolic Regression: Leveraging Large Language Models for Function Discoverycited by 31
  10. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 29
  11. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 25
  12. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
37 more
  1. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  2. 2025When can in-context learning generalize out of task distribution?cited by 16
  3. 2024On the Role of Depth and Looping for In-Context Learning with Task Diversitycited by 11
  4. 2025Solving Empirical Bayes via Transformerscited by 11
  5. 2024Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selectioncited by 9
  6. 2025Transformers Meet In-Context Learning: A Universal Approximation Theorycited by 6
  7. 2025On the Convergence of Gradient Descent on Learning Transformers with Residual Connectionscited by 6
  8. 2026TabFM: A Zero-Shot Foundation Model for Tabular Datacited by 6
  9. 2025In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Flycited by 5
  10. 2025Large Language Model Sourcing: A Surveycited by 5
  11. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  12. 2024Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networkscited by 3
  13. 2025Adversarially Pretrained Transformers May Be Universally Robust In-Context Learnerscited by 3
  14. 2025The Initialization Determines Whether In-Context Learning Is Gradient Descentcited by 3
  15. 2026Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledgecited by 3
  16. 2025Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspectivecited by 2
  17. 2026Learning to Adapt: In-Context Learning Beyond Stationaritycited by 2
  18. 2025Brewing Knowledge in Context: Distillation Perspectives on In-Context Learningcited by 1
  19. 2025On the Existence of Universal Simulators of Attentioncited by 1
  20. 2025Multi-Layer Attention is the Amplifier of Demonstration Effectivenesscited by 1
  21. 2025Linear Transformers Implicitly Discover Unified Numerical Algorithmscited by 1
  22. 2025How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPscited by 1
  23. 2025Optimal Attention Temperature Improves the Robustness of In-Context Learning under Distribution Shift in High Dimensionscited by 1
  24. 2026Transformers Learn Latent Mixture Models In-Context via Mirror Descentcited by 1
  25. 2026Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regressioncited by 1
  26. 2026Transformers Can Learn Posterior Predictive Distributions In-Contextcited by 1
  27. 2026Induction Heads Interpolate N-Gramscited by 1
  28. 2026Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatchcited by 1
  29. 2025In-Context Learning with Hypothesis-Class Guidancecited by 0
  30. 2026The Alchemy of Thought: Understanding In-Context Learning Through Supervised Classificationcited by 0
  31. 2026Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMscited by 0
  32. 2026Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICLcited by 0
  33. 2026In-Context Reward Adaptation for Robust Preference Modelingcited by 0
  34. 2026A Unified Framework for In-Context Learning with Causal and Masked Language Modelscited by 0
  35. 2026TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Modelscited by 0
  36. 2026Generalization of Transformer-Based Neural Quantum States via In-Context Learningcited by 0
  37. 2026PDE-PFN: Prior-Data Fitted Neural PDE Solvercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Toward generalizable learning of all (linear) first-order methods via memory augmented Transformers
  2. 2025Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression
  3. 2026Finite-Time Analysis of Gradient Descent for Shallow Transformers
  4. 2025How Transformers Learn In-Context Recall Tasks? Optimality, Training Dynamics and Generalization
  5. 2023The Transient Nature of Emergent In-Context Learning in Transformers
  6. 2026Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement
  7. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond
  8. 2026PowerStep: Memory-Efficient Adaptive Optimization via ℓp-Norm Steepest Descent