mlsanityData to 14 Oct 2026

On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures

arXiv 2410.11778 ↗First posted 15 Oct 2024cs.LG cs.IT math.ITcited by 15

Not on the map. Why?

Abstract (arXiv)

Although transformers have demonstrated impressive capabilities for in-context learning (ICL) in practice, theoretical understanding of the underlying mechanism that allows transformers to perform ICL is still in its infancy. This work aims to theoretically study the training dynamics of transformers for in-context classification tasks. We demonstrate that, for in-context classification of Gaussian mixtures under certain assumptions, a single-layer transformer trained via gradient descent converges to a globally optimal model at a linear rate. We further quantify the impact of the training and testing prompt lengths on the ICL inference error of the trained transformer. We show that when the lengths of training and testing prompts are sufficiently large, the prediction of the trained transformer approaches the ground truth distribution of the labels. Experimental results corroborate the theoretical findings.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 40 of its 51 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

40 of 51 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  8. 2022Transformers learn in-context by gradient descentcited by 732
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  11. 2014Convex Optimization: Algorithms and Complexitycited by 575
  12. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
28 more
  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  2. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  3. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  4. 2023Looped Transformers as Programmable Computerscited by 237
  5. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  6. 2022Language models show human-like content effects on reasoning taskscited by 173
  7. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  8. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  9. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
  10. 2023In-Context Convergence of Transformerscited by 115
  11. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  12. 2022Vision Transformers provably learn spatial structurecited by 93
  13. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 86
  14. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  15. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  16. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  17. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
  18. 2023Transformers as Support Vector Machinescited by 74
  19. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 74
  20. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  21. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 46
  22. 2024Dual Operating Modes of In-Context Learningcited by 43
  23. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  24. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 33
  25. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 25
  26. 2024Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannotcited by 22
  27. 2023Transformers can optimally learn regression mixture modelscited by 15
  28. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025When and How Unlabeled Data Provably Improve In-Context Learningcited by 6
  2. 2025Experience Replay Addresses Loss of Plasticity in Continual Learningcited by 4
  3. 2025Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformerscited by 3
  4. 2024Decision Feedback In-Context Symbol Detection over Block-Fading Channelscited by 2
  5. 2025Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regressioncited by 2
  6. 2026Unlabeled Data Can Provably Enhance In-Context Learning of Transformerscited by 2
  7. 2026Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learningcited by 2
  8. 2026Incremental Learning of Sparse Attention Patterns in Transformerscited by 2
  9. 2025Transformers are almost optimal metalearners for linear classificationcited by 1
  10. 2026Finite-Time Analysis of Gradient Descent for Shallow Transformerscited by 1
  11. 2025Decision Feedback In-Context Learning for Wireless Symbol Detectioncited by 0
  12. 2026Covariance-Aware Transformers for Quadratic Programming and Decision Makingcited by 0
3 more
  1. 2026Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMscited by 0
  2. 2026On the Capability and Limitation of Hard Promptcited by 0
  3. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0