mlsanityData to 7 Oct 2026

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?

arXiv 2310.08391 ↗First posted 12 Oct 2023stat.ML cs.LGcited by 93

Not on the map. Why?

Abstract (arXiv)

Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities, enabling them to solve unseen tasks solely based on input contexts without adjusting model parameters. In this paper, we study ICL in one of its simplest setups: pretraining a linearly parameterized single-layer linear attention model for linear regression with a Gaussian prior. We establish a statistical task complexity bound for the attention model pretraining, showing that effective pretraining only requires a small number of independent tasks. Furthermore, we prove that the pretrained model closely matches the Bayes optimal algorithm, i.e., optimally tuned ridge regression, by achieving nearly Bayes optimal risk on unseen tasks under a fixed context length. These theoretical findings complement prior experimental research and shed light on the statistical foundations of ICL.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

26 of 31 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  4. 2022Transformers learn in-context by gradient descentcited by 714
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  7. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  8. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  9. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  10. 2023Looped Transformers as Programmable Computerscited by 224
  11. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  12. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
14 more
  1. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  2. 2020Benign overfitting in ridge regressioncited by 152
  3. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  4. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  5. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  6. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 110
  7. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  8. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  9. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 67
  10. 2016Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regressioncited by 64
  11. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  12. 2021Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regressioncited by 37
  13. 2017A Markov Chain Theory Approach to Characterizing the Minimax Optimality of Stochastic Gradient Descent (for Least Squares)cited by 25
  14. 2023Finite-Sample Analysis of Learning High-Dimensional Single ReLU Neuroncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 41; 2025: 35; 2026: 15.

93 papers.

Sort:

By area:

  1. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  2. 2024Can large language models explore in-context?cited by 75
  3. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  4. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  5. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  6. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  7. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  8. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  9. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  10. 2024Dual Operating Modes of In-Context Learningcited by 42
  11. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  12. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
38 more
  1. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  2. 2024Iteration Head: A Mechanistic Study of Chain-of-Thoughtcited by 31
  3. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 30
  4. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  5. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 27
  6. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  7. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 22
  8. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21
  9. 2024Toward Understanding In-context vs. In-weight Learningcited by 19
  10. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  11. 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
  12. 2025In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learningcited by 17
  13. 2024On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixturescited by 15
  14. 2025Test-Time Training Provably Improves Transformers as In-context Learnerscited by 15
  15. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  16. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 14
  17. 2024Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysiscited by 14
  18. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 13
  19. 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 13
  20. 2024Vector-ICL: In-context Learning with Continuous Vector Representationscited by 13
  21. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 13
  22. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  23. 2024MLPs Learn In-Context on Regression and Classification Taskscited by 12
  24. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11
  25. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  26. 2025AI Agents as Universal Task Solverscited by 9
  27. 2023Transformers are Provably Optimal In-context Estimators for Wireless Communicationscited by 8
  28. 2025Understanding In-context Learning of Addition via Activation Subspacescited by 8
  29. 2025Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitationscited by 8
  30. 2025Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimationcited by 8
  31. 2024Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lenscited by 7
  32. 2025On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learningcited by 7
  33. 2024Context-Scaling versus Task-Scaling in In-Context Learningcited by 6
  34. 2025Analyzing limits for in-context learningcited by 6
  35. 2025Pretrain-Test Task Alignment Governs Generalization in In-Context Learningcited by 6
  36. 2024On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capabilitycited by 5
  37. 2024In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learningcited by 5
  38. 2024Re-examining learning linear functions in contextcited by 5

50 most cited of 93 All 93 (JSON)