mlsanityData to 14 Oct 2026

Transformer learns the cross-task prior and regularization for in-context learning

arXiv 2505.12138 ↗First posted 17 May 2025cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Transformers have shown a remarkable ability for in-context learning (ICL), making predictions based on contextual examples. However, while theoretical analyses have explored this prediction capability, the nature of the inferred context and its utility for downstream predictions remain open questions. This paper aims to address these questions by examining ICL for inverse linear regression (ILR), where context inference can be characterized by unsupervised learning of underlying weight vectors. Focusing on the challenging scenario of rank-deficient inverse problems, where context length is smaller than the number of unknowns in the weight vectors and regularization is necessary, we introduce a linear transformer to learn the inverse mapping from contextual examples to the underlying weight vector. Our findings reveal that the transformer implicitly learns both a prior distribution and an effective regularization strategy, outperforming traditional ridge regression and regularization methods. A key insight is the necessity of low task dimensionality relative to the context length for successful learning. Furthermore, we numerically verify that the error of the transformer estimator scales linearly with the noise level, the ratio of task dimension to context length, and the condition number of the input data. These results not only demonstrate the potential of transformers for solving ill-posed inverse problems, but also provide a new perspective towards understanding the knowledge extraction mechanism within transformers.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 17 May 2025.

Builds on

32 of 38 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  3. 2022Transformers learn in-context by gradient descentcited by 732
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  6. 2021Choose a Transformer: Fourier or Galerkincited by 343
  7. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  8. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  9. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  10. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  11. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  12. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
20 more
  1. 2023The Learnability of In-Context Learningcited by 127
  2. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  3. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  4. 2024Asymptotic theory of in-context learning by linear attentioncited by 55
  5. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  6. 2024Transformers are Universal In-context Learnerscited by 43
  7. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  8. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  9. 2023ViTO: Vision Transformer-Operatorcited by 29
  10. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  11. 2022Transformer Meets Boundary Value Inverse Problemscited by 22
  12. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  13. 2021Learning Regularization Parameters of Inverse Problems via Deep Neural Networkscited by 19
  14. 2024Bayesian scaling laws for in-context learningcited by 19
  15. 2024Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discoverycited by 18
  16. 2021A data-driven peridynamic continuum model for upscaling molecular dynamicscited by 17
  17. 2023Transformers can optimally learn regression mixture modelscited by 15
  18. 2023Deformable Cross-Attention Transformer for Medical Image Registrationcited by 9
  19. 2022To be or not to be stable, that is the question: understanding neural networks for inverse problemscited by 8
  20. 2023Ambiguity in solving imaging inverse problems with deep learning based operatorscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methodscited by 15
  2. 2025Learning Causal Graphs at Scale: A Foundation Model Approachcited by 4
  3. 2025Neural Interpretable PDEs: Harmonizing Fourier Insights with Attention for Scalable and Interpretable Physics Discoverycited by 2
  4. 2025An Attention-based Spatio-Temporal Neural Operator for Evolving Physicscited by 1
  5. 2026Transformers Learn Robust In-Context Regression under Distributional Uncertaintycited by 1
  6. 2026Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICLcited by 0
  7. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0