mlsanityData to 14 Oct 2026

Asymptotic Study of In-context Learning with Random Transformers through Equivalent Models

arXiv 2509.15152 ↗First posted 18 Sep 2025stat.ML cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We study the in-context learning (ICL) capabilities of pretrained Transformers in the setting of nonlinear regression. Specifically, we focus on a random Transformer with a nonlinear MLP head where the first layer is randomly initialized and fixed while the second layer is trained. Furthermore, we consider an asymptotic regime where the context length, input dimension, hidden dimension, number of training tasks, and number of training samples jointly grow. In this setting, we show that the random Transformer behaves equivalent to a finite-degree Hermite polynomial model in terms of ICL error. This equivalence is validated through simulations across varying activation functions, context lengths, hidden layer widths (revealing a double-descent phenomenon), and regularization settings. Our results offer theoretical and empirical insights into when and how MLP layers enhance ICL, and how nonlinearity and over-parameterization influence model performance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

25 of 34 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2022In-context Learning and Induction Headscited by 964
  5. 2022Transformers learn in-context by gradient descentcited by 732
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  8. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  9. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  10. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  12. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
13 more
  1. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  2. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  3. 2020Optimal Regularization Can Mitigate Double Descentcited by 109
  4. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  5. 2024Asymptotic theory of in-context learning by linear attentioncited by 55
  6. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 54
  7. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 46
  8. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  9. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  10. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 24
  11. 2025Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structurecited by 4
  12. 2023Optimal Nonlinearities Improve Generalization Performance of Random Featurescited by 3
  13. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Learning Linear Regression with Low-Rank Tasks in-Contextcited by 1
  2. 2025How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPscited by 1
  3. 2026Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regressioncited by 1