mlsanityData to 14 Oct 2026

Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective

arXiv 2505.14808 ↗First posted 20 May 2025stat.ML cs.LG math.STcited by 6

Not on the map. Why?

Abstract (arXiv)

The transformer's remarkable ability to perform in-context learning (ICL) has sparked a wide range of studies designed to understand its strengths and limitations. However, a theoretical understanding of when ICL can and cannot generalize beyond its pre-training data still remains unclear. This paper puts forth a minimal mathematical model that provably identifies when ICL can generalize out-of-distribution (OOD). By studying linear regression tasks parameterized with low-rank covariance matrices, we model distribution shifts as varying angles between subspaces and derive conditions under which a single-layer linear attention model interpolates across all angles. We show that if pre-training task vectors are drawn from a union of subspaces, transformers can generalize to all angle shifts--enabling ICL even in regions with zero probability mass in the training distribution. On the other hand, if the pre-training tasks are drawn from a single Gaussian, the test risk shows a non-negligible dependence on the angle, implying that ICL cannot generalize OOD. We empirically show that our results also hold for models such as GPT-2, and present experiments on how our results extend to nonlinear function classes.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

40 of 53 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2022Transformers learn in-context by gradient descentcited by 732
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  9. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  10. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  12. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
28 more
  1. 2022In-context Examples Selection for Machine Translationcited by 172
  2. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  3. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  4. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  5. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 145
  6. 2023In-Context Convergence of Transformerscited by 115
  7. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  8. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  9. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 65
  10. 2024Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributionscited by 53
  11. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 51
  12. 2023Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscited by 45
  13. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 43
  14. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  15. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  16. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 33
  17. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  18. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  19. 2023A Closer Look at In-Context Learning under Distribution Shiftscited by 25
  20. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 24
  21. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  22. 2024Retrieval-style In-Context Learning for Few-shot Hierarchical Text Classificationcited by 16
  23. 2025Test-Time Training Provably Improves Transformers as In-context Learnerscited by 16
  24. 2024A Global Geometric Analysis of Maximal Coding Rate Reductioncited by 13
  25. 2025An Overview of Low-Rank Structures in the Training and Adaptation of Large Modelscited by 9
  26. 2025Rethinking Invariance in In-context Learningcited by 8
  27. 2024Improving In-Context Learning with Prediction Feedback for Sentiment Analysiscited by 4
  28. 2025Provable Benefits of Task-Specific Prompts for In-context Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Transformers Meet In-Context Learning: A Universal Approximation Theorycited by 6
  2. 2025Learning Linear Regression with Low-Rank Tasks in-Contextcited by 1
  3. 2026The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspacescited by 1
  4. 2026A Unified Framework for In-Context Learning with Causal and Masked Language Modelscited by 0
  5. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0
  6. 2026Retrieval-Based In-Context Learning: A Domain Adaptation Frameworkcited by 0