mlsanityData to 14 Oct 2026

Test time training enhances in-context learning of nonlinear functions

arXiv 2509.25741 ↗First posted 30 Sep 2025stat.ML cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction to adapt to the test data. While TTT has demonstrated considerable empirical success, its theoretical underpinnings remain limited, particularly for nonlinear models. In this paper, we investigate the combination of TTT with in-context learning (ICL), where the model is given a few examples from the target distribution at inference time. We analyze this framework in the setting of single-index models, where the feature vector is drawn from a hidden low-dimensional subspace. For single-layer transformers trained with gradient-based algorithms and adopting TTT, we establish an upper bound on the prediction risk. Our theory reveals that TTT enables the single-layer transformers to adapt to both the feature vector and the link function, which vary across tasks. This creates a sharp contrast with ICL alone, which is theoretically difficult to adapt to shifts in the link function. Moreover, we provide the convergence rate with respect to the data length, showing the predictive error can be driven arbitrarily close to the noise level as the context size and the network width grow.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

36 of 46 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Transformers learn in-context by gradient descentcited by 732
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  4. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  5. 2024Many-Shot In-Context Learningcited by 206
  6. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  7. 2019Linearized two-layers neural networks in high dimensioncited by 155
  8. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  9. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  10. 2025Test-Time Training Done Rightcited by 130
  11. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  12. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
24 more
  1. 2023In-Context Convergence of Transformerscited by 115
  2. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 96
  3. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  4. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  5. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  6. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 67
  7. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 65
  8. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  9. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  10. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 61
  11. 2023Learning time-scales in two-layers neural networkscited by 57
  12. 2024Transformers Provably Solve Parity Efficiently with Chain of Thoughtcited by 50
  13. 2025Test-Time Learning for Large Language Modelscited by 50
  14. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 46
  15. 2024Computational-Statistical Gaps in Gaussian Single-Index Modelscited by 44
  16. 2025Learning without training: The implicit dynamics of in-context learningcited by 43
  17. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 37
  18. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 33
  19. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  20. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  21. 2025Test-Time Training Provably Improves Transformers as In-context Learnerscited by 16
  22. 2024On the Complexity of Learning Sparse Functions with Statistical and Gradient Queriescited by 15
  23. 2025Analyzing limits for in-context learningcited by 6
  24. 2025The Inherent Limits of Pretrained LLMs: The Unexpected Convergence of Instruction Tuning and In-Context Learning Capabilitiescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Efficient and Minimax Optimal In-context Nonparametric Regression with Transformerscited by 7
  2. 2025Optimal In-context Adaptivity and Distributional Robustness of Transformerscited by 4
  3. 2026A Decision-Theoretic View of Test-Time Training: When, How Far, and Which Directions to Adaptcited by 0