mlsanityData to 14 Oct 2026

Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection

arXiv 2306.04637 ↗First posted 7 Jun 2023cs.LG cs.AI cs.CLcited by 298

Not on the map. Why?

Cited by 22 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural sequence models based on the transformer architecture have demonstrated remarkable in-context learning (ICL) abilities, where they can perform new tasks when prompted with training and test examples, without any parameter update to the model. This work first provides a comprehensive statistical theory for transformers to perform ICL. Concretely, we show that transformers can implement a broad class of standard machine learning algorithms in context, such as least squares, ridge regression, Lasso, learning generalized linear models, and gradient descent on two-layer neural networks, with near-optimal predictive power on various in-context data distributions. Using an efficient implementation of in-context gradient descent as the underlying mechanism, our transformer constructions admit mild size bounds, and can be learned with polynomially many pretraining sequences. Building on these ``base'' ICL algorithms, intriguingly, we show that transformers can implement more complex ICL procedures involving in-context algorithm selection, akin to what a statistician can do in real life -- A single transformer can adaptively select different base ICL algorithms -- or even perform qualitatively different tasks -- on different input sequences, without any explicit prompting of the right algorithm or task. We both establish this in theory by explicit constructions, and also observe this phenomenon experimentally. In theory, we construct two general mechanisms for algorithm selection with concrete examples: pre-ICL testing, and post-ICL validation. As an example, we use the post-ICL validation mechanism to construct a transformer that can perform nearly Bayes-optimal ICL on a challenging task -- noisy linear models with mixed noise levels. Experimentally, we demonstrate the strong in-context algorithm selection capabilities of standard transformer architectures.

Key facts. Cited by 298 papers in the map (data through 14 Oct 2026). Builds on 41 of its 97 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

41 of 97 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2016Layer Normalizationcited by 5,669
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  12. 2022A Generalist Agentcited by 808
29 more
  1. 2022Transformers learn in-context by gradient descentcited by 732
  2. 2022A Survey on In-context Learningcited by 721
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  5. 2021MetaICL: Learning to Learn In Contextcited by 454
  6. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  7. 2023Larger language models do in-context learning differentlycited by 394
  8. 2022Transformers Learn Shortcuts to Automatacited by 268
  9. 2023Looped Transformers as Programmable Computerscited by 237
  10. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  11. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  12. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  13. 2016Learning to Optimizecited by 195
  14. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  15. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  16. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  17. 2020ProGen: Language Modeling for Protein Generationcited by 128
  18. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  19. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  20. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  21. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  22. 2022Vision Transformers provably learn spatial structurecited by 93
  23. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  24. 2023A Study on ReLU and Softmax in Transformercited by 53
  25. 2021Approximating How Single Head Attention Learnscited by 38
  26. 2017A Simple Neural Attentive Meta-Learnercited by 23
  27. 2018Incremental Learning-to-Learn with Statistical Guaranteescited by 16
  28. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 16
  29. 2020On the Ability and Limitations of Transformers to Recognize Formal Languagescited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 26; 2024: 114; 2025: 84; 2026: 74.

298 papers.

Sort:

By area:

  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  2. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  3. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  4. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
  5. 2023In-Context Convergence of Transformerscited by 115
  6. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  7. 2023The Expressive Power of Low-Rank Adaptationcited by 102
  8. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  9. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  10. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 88
  11. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  12. 2024Can large language models explore in-context?cited by 76
38 more
  1. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
  2. 2023Transformers as Support Vector Machinescited by 74
  3. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 65
  4. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  5. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  6. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 57
  7. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 57
  8. 2024Asymptotic theory of in-context learning by linear attentioncited by 55
  9. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  10. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 49
  11. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 49
  12. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 46
  13. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  14. 2024Is Mamba Capable of In-Context Learning?cited by 45
  15. 2024Why Larger Language Models Do In-context Learning Differently?cited by 45
  16. 2023Replacing softmax with ReLU in Vision Transformerscited by 44
  17. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  18. 2024Dual Operating Modes of In-Context Learningcited by 43
  19. 2025Learning without training: The implicit dynamics of in-context learningcited by 43
  20. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 40
  21. 2023Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Modelscited by 39
  22. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 39
  23. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  24. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  25. 2023Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selectioncited by 37
  26. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 37
  27. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 37
  28. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  29. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  30. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 34
  31. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 33
  32. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  33. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  34. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  35. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  36. 2024Separations in the Representational Capabilities of Transformers and Recurrent Architecturescited by 29
  37. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 29
  38. 2024Linear Transformers are Versatile In-Context Learnerscited by 28

50 most cited of 298 All 298 (JSON)