mlsanityData to 14 Oct 2026

Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methods

arXiv 2506.10959 ↗First posted 12 Jun 2025cs.LG cs.AI math.STcited by 15

Not on the map. Why?

Abstract (arXiv)

While in-context learning (ICL) has achieved remarkable success in natural language and vision domains, its theoretical understanding-particularly in the context of structured geometric data-remains unexplored. This paper initiates a theoretical study of ICL for regression of H"older functions on manifolds. We establish a novel connection between the attention mechanism and classical kernel methods, demonstrating that transformers effectively perform kernel-based prediction at a new query through its interaction with the prompt. This connection is validated by numerical experiments, revealing that the learned query-prompt scores for H"older functions are highly correlated with the Gaussian kernel. Building on this insight, we derive generalization error bounds in terms of the prompt length and the number of training tasks. When a sufficient number of training tasks are observed, transformers give rise to the minimax regression rate of H"older functions on manifolds, which scales exponentially with respect to the prompt length with the exponent depending on the intrinsic dimension of the manifold, rather than the ambient space dimension. Our result also characterizes how the generalization error scales with the number of training tasks, shedding light on the complexity of transformers as in-context kernel algorithm learners. Our findings provide foundational insights into the role of geometry in ICL and novels tools to study ICL of nonlinear models.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 34 of its 60 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

34 of 60 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  6. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  7. 2022Transformers learn in-context by gradient descentcited by 732
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  10. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  11. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  12. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
22 more
  1. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  2. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  3. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 213
  4. 2023A mathematical perspective on Transformerscited by 164
  5. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 101
  6. 2019Deep ReLU network approximation of functions on a manifoldcited by 74
  7. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  8. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  9. 2015Provable approximation properties for deep neural networkscited by 60
  10. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 37
  11. 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networkscited by 32
  12. 2023Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Inputcited by 32
  13. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  14. 2023Understanding Emergent In-Context Learning from a Kernel Regression Perspectivecited by 24
  15. 2020Distribution Approximation and Statistical Estimation Guarantees of Generative Adversarial Networkscited by 23
  16. 2021Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machinescited by 22
  17. 2024Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discoverycited by 18
  18. 2021On the rate of convergence of a classifier based on a Transformer encodercited by 16
  19. 2020Implicit Kernel Attentioncited by 13
  20. 2016Adaptive Geometric Multiscale Approximations for Intrinsically Low-dimensional Datacited by 8
  21. 2025Transformer learns the cross-task prior and regularization for in-context learningcited by 7
  22. 2025Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insightscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 13.

15 papers.

Sort:

By area:

  1. 2025Pretrain-Test Task Alignment Governs Generalization in In-Context Learningcited by 8
  2. 2026Efficient and Minimax Optimal In-context Nonparametric Regression with Transformerscited by 7
  3. 2026Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizercited by 3
  4. 2026Learn to Evolve: Self-supervised Neural JKO Operator for Wasserstein Gradient Flowcited by 2
  5. 2026Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unitycited by 2
  6. 2026In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspectivecited by 2
  7. 2026Beyond the Prompt in Large Language Models: Comprehension, In-Context Learning, and Chain-of-Thoughtcited by 1
  8. 2026Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regressioncited by 1
  9. 2026Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalizationcited by 1
  10. 2026Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatchcited by 1
  11. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1
  12. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 1
3 more
  1. 2025In-Context Multi-Operator Learning with DeepOSetscited by 0
  2. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0
  3. 2026Transformer-based In-Context Score Operator Learning for Diffusion Modelscited by 0