mlsanityData to 14 Oct 2026

Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers

arXiv 2601.15014 ↗First posted 21 Jan 2026stat.ML cs.LG math.STcited by 7

Not on the map. Why?

Abstract (arXiv)

We study in-context learning for nonparametric regression with α-H"older smooth regression functions, for some α>0. We prove that, with n in-context examples and d-dimensional regression covariates, a pretrained transformer with Theta(log n) parameters and Omegabigl(n2α/(2α+d)log3 nbigr) pretraining sequences can achieve the minimax optimal rate of convergence Obigl(n-2α/(2α+d)bigr) in mean squared error. Our result requires substantially fewer transformer parameters and pretraining sequences than previous results in the literature. This is achieved by showing that transformers are able to approximate local polynomial estimators efficiently by implementing a kernel-weighted polynomial basis and then running gradient descent.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 21 Jan 2026.

Builds on

33 of 46 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  9. 2022Transformers learn in-context by gradient descentcited by 732
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  12. 2014Convex Optimization: Algorithms and Complexitycited by 575
21 more
  1. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  2. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  3. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  4. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  5. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  6. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  7. 2020Deep Network Approximation for Smooth Functionscited by 164
  8. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  9. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  10. 2023Statistical Foundations of Prior-Data Fitted Networkscited by 65
  11. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  12. 2024Asymptotic theory of in-context learning by linear attentioncited by 55
  13. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 37
  14. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  15. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  16. 2025In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learningcited by 22
  17. 2025Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methodscited by 15
  18. 2025Task Generalization With AutoRegressive Compositional Structure: Can Learning From D Tasks Generalize to DT Tasks?cited by 11
  19. 2025A Theoretical Framework for Prompt Engineering: Approximating Smooth Functions with Transformer Promptscited by 6
  20. 2025Optimal In-context Adaptivity and Distributional Robustness of Transformerscited by 4
  21. 2025Test time training enhances in-context learning of nonlinear functionscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Propagation of Chaos in Contextual Flow Mapscited by 3
  2. 2026Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learningcited by 2
  3. 2026Beyond the Prompt in Large Language Models: Comprehension, In-Context Learning, and Chain-of-Thoughtcited by 1
  4. 2026Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversitycited by 1
  5. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1
  6. 2026A Unified Framework for In-Context Learning with Causal and Masked Language Modelscited by 0
  7. 2026Vector Search As Nearest Neighbor Matching: RAG-based Policy Learning in Causal Inferencecited by 0