mlsanityData to 7 Oct 2026

What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

arXiv 2208.01066 ↗First posted 1 Aug 2022cs.CL cs.LGcited by 700

Not on the map. Why?

Cited by 43 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In-context learning refers to the ability of a model to condition on a prompt sequence consisting of in-context examples (input-output pairs corresponding to some task) along with a new query input, and generate the corresponding output. Crucially, in-context learning happens only at inference time without any parameter updates to the model. While large language models such as GPT-3 exhibit some ability to perform in-context learning, it is unclear what the relationship is between tasks on which this succeeds and what is present in the training data. To make progress towards understanding in-context learning, we consider the well-defined problem of training a model to in-context learn a function class (e.g., linear functions): that is, given data derived from some functions in the class, can we train a model to in-context learn "most" functions from this class? We show empirically that standard Transformers can be trained from scratch to perform in-context learning of linear functions -- that is, the trained model is able to learn unseen linear functions from in-context examples with performance comparable to the optimal least squares estimator. In fact, in-context learning is possible even under two forms of distribution shift: (i) between the training data of the model and inference-time prompts, and (ii) between the in-context examples and the query input during inference. We also show that we can train Transformers to in-context learn more complex function classes -- namely sparse linear functions, two-layer neural networks, and decision trees -- with performance that matches or exceeds task-specific learning algorithms. Our code and models are available at https://github.com/dtsip/in-context-learning .

Key facts. Cited by 700 papers in the map (data through 7 Oct 2026). Builds on 53 of its 85 references that are in the map. First posted on arXiv 1 Aug 2022.

Builds on

53 of 85 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
38 more
  1. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  4. 2015Pointer Networkscited by 882
  5. 2018Image Transformercited by 770
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  8. 2018Universal Transformerscited by 613
  9. 2016Neural Combinatorial Optimization with Reinforcement Learningcited by 600
  10. 2017Learning Combinatorial Optimization Algorithms over Graphscited by 557
  11. 2021MetaICL: Learning to Learn In Contextcited by 450
  12. 2018Conditional Neural Processescited by 421
  13. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  14. 2021Transformers Can Do Bayesian Inferencecited by 338
  15. 2018Neural Processescited by 307
  16. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  17. 2019Attentive Neural Processescited by 280
  18. 2022Can language models learn from explanations in context?cited by 248
  19. 2018Learning a SAT Solver from Single-Bit Supervisioncited by 198
  20. 2016Learning to Optimizecited by 195
  21. 2021Thinking Like Transformerscited by 182
  22. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  23. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  24. 2021Meta-learning via Language Model In-context Tuningcited by 163
  25. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  26. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  27. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  28. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  29. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  30. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  31. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 108
  32. 2020When Do Curricula Work?cited by 105
  33. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  34. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  35. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  36. 2020Meta Learning Backpropagation And Improving Itcited by 49
  37. 2021Approximating How Single Head Attention Learnscited by 38
  38. 2019More Data Can Hurt for Linear Regression: Sample-wise Double Descentcited by 37

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 123; 2024: 199; 2025: 200; 2026: 161.

700 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023Large Language Modelscited by 774
  3. 2022A Survey on In-context Learningcited by 717
  4. 2022Transformers learn in-context by gradient descentcited by 714
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  7. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  8. 2023Larger language models do in-context learning differentlycited by 386
  9. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  10. 2023In-Context Learning Creates Task Vectorscited by 302
  11. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  12. 2023Function Vectors in Large Language Modelscited by 292
38 more
  1. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  2. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
  3. 2023Looped Transformers as Programmable Computerscited by 224
  4. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  5. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  6. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  7. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  8. 2024Many-Shot In-Context Learningcited by 203
  9. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  10. 2023Large Language Models as General Pattern Machinescited by 196
  11. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  12. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  13. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  14. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 143
  15. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  16. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  17. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  18. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  19. 2023The Learnability of In-Context Learningcited by 126
  20. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  21. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  22. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  23. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  24. 2023In-Context Convergence of Transformerscited by 112
  25. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  26. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  27. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  28. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  29. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  30. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  31. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  32. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  33. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  34. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  35. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  36. 2023In-Context Learning through the Bayesian Prismcited by 78
  37. 2024Can large language models explore in-context?cited by 75
  38. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74

50 most cited of 700 All 700 (JSON)