mlsanityData to 7 Oct 2026

Learning Linear Regression with Low-Rank Tasks in-Context

arXiv 2510.04548 ↗First posted 6 Oct 2025cond-mat.dis-nn cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) is a key building block of modern large language models, yet its theoretical mechanisms remain poorly understood. It is particularly mysterious how ICL operates in real-world applications where tasks have a common structure. In this work, we address this problem by analyzing a linear attention model trained on low-rank regression tasks. Within this setting, we precisely characterize the distribution of predictions and the generalization error in the high-dimensional limit. Moreover, we find that statistical fluctuations in finite pre-training data induce an implicit regularization. Finally, we identify a sharp phase transition of the generalization error governed by task structure. These results provide a framework for understanding how transformers learn to learn the task structure.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

24 of 37 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2022Transformers learn in-context by gradient descentcited by 714
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  8. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  9. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  10. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  11. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  12. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
12 more
  1. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 72
  2. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  3. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  4. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  5. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  6. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  7. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntaxcited by 22
  8. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 14
  9. 2025Pretrain-Test Task Alignment Governs Generalization in In-Context Learningcited by 6
  10. 2025Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivecited by 5
  11. 2025Summary statistics of learning link changing neural representations to behaviorcited by 4
  12. 2025Asymptotic Study of In-context Learning with Random Transformers through Equivalent Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026An Asymptotic Theory of Chain-of-Thought in In-Context Learningcited by 0