mlsanityData to 7 Oct 2026

Transformers as Algorithms: Generalization and Stability in In-context Learning

arXiv 2301.07067 ↗First posted 17 Jan 2023cs.LG cs.CL stat.MLcited by 220

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) is a type of prompting where a transformer model operates on a sequence of (input, output) examples and performs inference on-the-fly. In this work, we formalize in-context learning as an algorithm learning problem where a transformer model implicitly constructs a hypothesis function at inference-time. We first explore the statistical aspects of this abstraction through the lens of multitask learning: We obtain generalization bounds for ICL when the input prompt is (1) a sequence of i.i.d. (input, label) pairs or (2) a trajectory arising from a dynamical system. The crux of our analysis is relating the excess risk to the stability of the algorithm implemented by the transformer. We characterize when transformer/attention architecture provably obeys the stability condition and also provide empirical verification. For generalization on unseen tasks, we identify an inductive bias phenomenon in which the transfer learning risk is governed by the task complexity and the number of MTL tasks in a highly predictable manner. Finally, we provide numerical evaluations that (1) demonstrate transformers can indeed implement near-optimal algorithms on classical regression problems with i.i.d. and dynamic data, (2) provide insights on stability, and (3) verify our theoretical predictions.

Key facts. Cited by 220 papers in the map (data through 7 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 17 Jan 2023.

Builds on

44 of 67 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2022In-context Learning and Induction Headscited by 930
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  7. 2022Transformers learn in-context by gradient descentcited by 714
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  10. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  11. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  12. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
32 more
  1. 2015The Benefit of Multitask Representation Learningcited by 194
  2. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  3. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  4. 2020Provable Meta-Learning of Linear Representationscited by 161
  5. 2018Learning Without Mixing: Towards A Sharp Analysis of Linear System Identificationcited by 158
  6. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 151
  7. 2016A vector-contraction inequality for Rademacher complexitiescited by 141
  8. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  9. 2018Near optimal finite time identification of arbitrary linear dynamical systemscited by 89
  10. 2020Active Learning for Nonlinear System Identification with Guaranteescited by 78
  11. 2020On the Value of Target Data in Transfer Learningcited by 62
  12. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  13. 2022Statistical Learning Theory for Control: A Finite Sample Perspectivecited by 52
  14. 2020Meta Learning Backpropagation And Improving Itcited by 49
  15. 2020Learning nonlinear dynamical systems from a single trajectorycited by 46
  16. 2020Meta-learning for mixed linear regressioncited by 44
  17. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  18. 2020Non-asymptotic and Accurate Learning of Nonlinear Dynamical Systemscited by 39
  19. 2022Learning with little mixingcited by 33
  20. 2022MAML and ANIL Provably Learn Representationscited by 30
  21. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
  22. 2022Single Trajectory Nonparametric Learning of Nonlinear Dynamicscited by 26
  23. 2022Multi-Task Imitation Learning for Linear Dynamical Systemscited by 22
  24. 2022Towards Sample-efficient Overparameterized Meta-learningcited by 21
  25. 2019A Tutorial on Concentration Bounds for System Identificationcited by 19
  26. 2021Joint Learning of Linear Time-Invariant Dynamical Systemscited by 16
  27. 2022Provable and Efficient Continual Representation Learningcited by 14
  28. 2022Non-Stationary Representation Learning in Sequential Linear Banditscited by 13
  29. 2015Regularized linear system identification using atomic, nuclear and kernel-based norms: the role of the stability constraintcited by 11
  30. 2022Understanding Benign Overfitting in Gradient-Based Meta Learningcited by 11
  31. 2022Generalization Properties of Retrieval-based Modelscited by 5
  32. 2022Joint Learning-Based Stabilization of Multiple Unknown Linear Systemscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 32; 2024: 82; 2025: 70; 2026: 36.

220 papers.

Sort:

By area:

  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  2. 2023Function Vectors in Large Language Modelscited by 292
  3. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  4. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  5. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  6. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  7. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  8. 2023In-Context Convergence of Transformerscited by 112
  9. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  10. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  11. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  12. 2024Can large language models explore in-context?cited by 75
38 more
  1. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  2. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  3. 2023Transformers as Support Vector Machinescited by 67
  4. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  5. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 62
  6. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  7. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  8. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  9. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 47
  10. 2023Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscited by 45
  11. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  12. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  13. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 43
  14. 2024Dual Operating Modes of In-Context Learningcited by 42
  15. 2023Memorization Capacity of Multi-Head Attention in Transformerscited by 39
  16. 2024An Information-Theoretic Analysis of In-Context Learningcited by 39
  17. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 39
  18. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 38
  19. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  20. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  21. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  22. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 36
  23. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  24. 2023How are Prompts Different in Terms of Sensitivity?cited by 34
  25. 2024From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformerscited by 34
  26. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  27. 2024Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Modelscited by 34
  28. 2024Large Language Models as Markov Chainscited by 33
  29. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  30. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  31. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 30
  32. 2023Large Language Model for Multi-objective Evolutionary Optimizationcited by 29
  33. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  34. 2023Understanding In-Context Learning from Repetitionscited by 27
  35. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 26
  36. 2023A Closer Look at In-Context Learning under Distribution Shiftscited by 25
  37. 2024Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Modelscited by 25
  38. 2024Zero-shot forecasting of chaotic systemscited by 25

50 most cited of 220 All 220 (JSON)