mlsanityData to 14 Oct 2026

Pretrained transformer efficiently learns low-dimensional target functions in-context

arXiv 2411.02544 ↗First posted 4 Nov 2024cs.LG stat.MLcited by 31

Not on the map. Why?

Abstract (arXiv)

Transformers can efficiently learn in-context from example demonstrations. Most existing theoretical analyses studied the in-context learning (ICL) ability of transformers for linear function classes, where it is typically shown that the minimizer of the pretraining loss implements one gradient descent step on the least squares objective. However, this simplified linear setting arguably does not demonstrate the statistical efficiency of ICL, since the pretrained transformer does not outperform directly solving linear regression on the test prompt. In this paper, we study ICL of a nonlinear function class via transformer with nonlinear MLP layer: given a class of textitsingle-index target functions f_*(boldsymbolx) = σ_*(langleboldsymbolx,boldsymbolβrangle), where the index features boldsymbolβ∈ℝd are drawn from a r-dimensional subspace, we show that a nonlinear transformer optimized by gradient descent (with a pretraining sample complexity that depends on the textitinformation exponent of the link functions σ_*) learns f_* in-context with a prompt length that only depends on the dimension of the distribution of target functions r; in contrast, any algorithm that directly learns f_* on test prompt yields a statistical complexity that scales with the ambient dimension d. Our result highlights the adaptivity of the pretrained transformer to low-dimensional structures of the function class, which enables sample-efficient ICL that outperforms estimators that only have access to the in-context data.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 55 of its 64 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

55 of 64 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Transformers learn in-context by gradient descentcited by 732
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  6. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  7. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  8. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  9. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  10. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  11. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  12. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
38 more
  1. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  2. 2019Linearized two-layers neural networks in high dimensioncited by 155
  3. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  4. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  5. 2016A vector-contraction inequality for Rademacher complexitiescited by 142
  6. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  7. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  8. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  9. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
  10. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
  11. 2023In-Context Convergence of Transformerscited by 115
  12. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  13. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  14. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  15. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  16. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
  17. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 74
  18. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  19. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 65
  20. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  21. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  22. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 61
  23. 2023Learning time-scales in two-layers neural networkscited by 57
  24. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 57
  25. 2024Asymptotic theory of in-context learning by linear attentioncited by 55
  26. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 52
  27. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 50
  28. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 46
  29. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 46
  30. 2024An Information-Theoretic Analysis of In-Context Learningcited by 40
  31. 2017Fundamental Limits of Weak Recovery with Applications to Phase Retrievalcited by 39
  32. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 37
  33. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 37
  34. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  35. 2020Learning Polynomials of Few Relevant Dimensionscited by 34
  36. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 33
  37. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  38. 2023Gradient-Based Feature Learning under Structured Datacited by 32

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 11.

31 papers.

Sort:

By area:

  1. 2025In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learningcited by 22
  2. 2025Provable Failure of Language Models in Learning Majority Boolean Logic via Gradient Descentcited by 8
  3. 2025On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learningcited by 7
  4. 2025Transformer learns the cross-task prior and regularization for in-context learningcited by 7
  5. 2026Efficient and Minimax Optimal In-context Nonparametric Regression with Transformerscited by 7
  6. 2025Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivecited by 6
  7. 2025From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformerscited by 6
  8. 2025CoT-ICL Lab: A Synthetic Framework for Studying Chain-of-Thought Learning from In-Context Demonstrationscited by 4
  9. 2025Provable In-Context Learning of Nonlinear Regression with Transformerscited by 4
  10. 2025Optimal In-context Adaptivity and Distributional Robustness of Transformerscited by 4
  11. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 3
  12. 2025Asymptotic Study of In-context Learning with Random Transformers through Equivalent Modelscited by 3
19 more
  1. 2025Test time training enhances in-context learning of nonlinear functionscited by 3
  2. 2025Neural Interpretable PDEs: Harmonizing Fourier Insights with Attention for Scalable and Interpretable Physics Discoverycited by 2
  3. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  4. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
  5. 2025Multi-Layer Attention is the Amplifier of Demonstration Effectivenesscited by 1
  6. 2025Learning Linear Regression with Low-Rank Tasks in-Contextcited by 1
  7. 2025Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learningcited by 1
  8. 2025Transformers are almost optimal metalearners for linear classificationcited by 1
  9. 2025How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPscited by 1
  10. 2026Bayesian Optimality of In-Context Learning with Selective State Spacescited by 1
  11. 2026The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledgecited by 1
  12. 2026Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatchcited by 1
  13. 2026Reconciling In-Context and In-Weight Learning via Dual Representation Space Encodingcited by 0
  14. 2026Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICLcited by 0
  15. 2026Vector Search As Nearest Neighbor Matching: RAG-based Policy Learning in Causal Inferencecited by 0
  16. 2026Handover of In-Context Learning State Across Session Boundariescited by 0
  17. 2026Adaptive Mean Estimation by In-Context Learning: A Gradient-Flow Analysiscited by 0
  18. 2026What can linear attention learn from nonlinear teachers in-context?cited by 0
  19. 2026In-context classification: Calibration and noise in linear attention with softmax readoutcited by 0