mlsanityData to 14 Oct 2026

A Closer Look at In-Context Learning under Distribution Shifts

arXiv 2305.16704 ↗First posted 26 May 2023cs.LG stat.MLcited by 25

Not on the map. Why?

Abstract (arXiv)

In-context learning, a capability that enables a model to learn from input examples on the fly without necessitating weight updates, is a defining characteristic of large language models. In this work, we follow the setting proposed in (Garg et al., 2022) to better understand the generality and limitations of in-context learning from the lens of the simple yet fundamental task of linear regression. The key question we aim to address is: Are transformers more adept than some natural and simpler architectures at performing in-context learning under varying distribution shifts? To compare transformers, we propose to use a simple architecture based on set-based Multi-Layer Perceptrons (MLPs). We find that both transformers and set-based MLPs exhibit in-context learning under in-distribution evaluations, but transformers more closely emulate the performance of ordinary least squares (OLS). Transformers also display better resilience to mild distribution shifts, where set-based MLPs falter. However, under severe distribution shifts, both models' in-context learning abilities diminish.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

15 of 21 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Deep Setscited by 1,421
  4. 2022In-context Learning and Induction Headscited by 964
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2022Transformers learn in-context by gradient descentcited by 732
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  9. 2023Larger language models do in-context learning differentlycited by 394
  10. 2022Transformers Learn Shortcuts to Automatacited by 268
  11. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  12. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
3 more
  1. 2016Discovering Causal Signals in Imagescited by 79
  2. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 62
  3. 2022Transformers generalize differently from information stored in context vs in weightscited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 11; 2025: 4; 2026: 5.

25 papers.

Sort:

By area:

  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  2. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  3. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  4. 2024On the Out-Of-Distribution Generalization of Multimodal Large Language Modelscited by 30
  5. 2024Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Datacited by 25
  6. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  7. 2023How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigationcited by 22
  8. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  9. 2025When can in-context learning generalize out of task distribution?cited by 16
  10. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 14
  11. 2025Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivecited by 6
  12. 2024On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capabilitycited by 5
13 more
  1. 2025Understanding the Generalization of In-Context Learning in Transformers: An Empirical Studycited by 4
  2. 2023Positional Information Matters for Invariant In-Context Learning: A Case Study of Simple Function Classescited by 3
  3. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
  4. 2024In-context Learning in Presence of Spurious Correlationscited by 3
  5. 2023Context is Environmentcited by 1
  6. 2024Generalization vs. Specialization under Concept Shiftcited by 1
  7. 2026The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspacescited by 1
  8. 2024Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Taskscited by 0
  9. 2025Technical Debt in In-Context Learning: Diminishing Efficiency in Long Contextcited by 0
  10. 2026Reconciling In-Context and In-Weight Learning via Dual Representation Space Encodingcited by 0
  11. 2026Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classificationcited by 0
  12. 2026Retrieval-Guided Fine-Tuning as Noisy Estimation: Risk bounds and Architectural Analysiscited by 0
  13. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0