mlsanityData to 7 Oct 2026

Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer

arXiv 2605.05176 ↗First posted 6 May 2026cs.LG cs.NA math.NAcited by 2

Not on the map. Why?

Abstract (arXiv)

Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite its demonstrated efficacy across various domains, the theoretical understanding of ICL is still developing. Whereas most existing theory has focused on linear models, we study ICL in the nonlinear regression setting. Through the interaction mechanism in attention, we explicitly construct transformer networks to realize nonlinear features, such as polynomial or spline bases, which span a wide class of functions. Based on this construction, we establish a framework to analyze end-to-end in-context nonlinear regression with the constructed features. Our theory provides finite-sample generalization error bounds in terms of context length and training set size. We numerically validate the theory on synthetic regression tasks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

32 of 38 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Transformers learn in-context by gradient descentcited by 714
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  4. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  5. 2016Error bounds for approximations with deep ReLU networkscited by 567
  6. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  7. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  8. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  9. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  10. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  11. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  12. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
20 more
  1. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  2. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  3. 2023In-Context Convergence of Transformerscited by 112
  4. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  5. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  6. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  7. 2023Length Generalization in Arithmetic Transformerscited by 64
  8. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  9. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  10. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  11. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
  12. 2023Understanding Addition in Transformerscited by 34
  13. 2023Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Inputcited by 32
  14. 2025Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Modelscited by 31
  15. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  16. 2025Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methodscited by 13
  17. 2025On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learningcited by 7
  18. 2025Analyzing limits for in-context learningcited by 6
  19. 2025Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insightscited by 3
  20. 2024Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignmentcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1
  2. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0