mlsanityData to 7 Oct 2026

In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation

arXiv 2502.08136 ↗First posted 12 Feb 2025cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper investigates approximation-theoretic aspects of the in-context learning capability of the transformers in representing a family of noisy linear dynamical systems. Our first theoretical result establishes an upper bound on the approximation error of multi-layer transformers with respect to an L2-testing loss uniformly defined across tasks. This result demonstrates that transformers with logarithmic depth can achieve error bounds comparable with those of the least-squares estimator. In contrast, our second result establishes a non-diminishing lower bound on the approximation error for a class of single-layer linear transformers, which suggests a depth-separation phenomenon for transformers in the in-context learning of dynamical systems. Moreover, this second result uncovers a critical distinction in the approximation power of single-layer linear transformers when learning from IID versus non-IID data.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 12 Feb 2025.

Builds on

45 of 62 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2021Transformers in Vision: A Surveycited by 1,009
  5. 2022In-context Learning and Induction Headscited by 930
  6. 2022Transformers learn in-context by gradient descentcited by 714
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2021Transformers Can Do Bayesian Inferencecited by 338
  10. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  11. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  12. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
33 more
  1. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  2. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  3. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  4. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
  5. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  6. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  7. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  8. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  9. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  10. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  11. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  12. 2020Learning nonlinear dynamical systems from a single trajectorycited by 46
  13. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 46
  14. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  15. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  16. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
  17. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 30
  18. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  19. 2024Transformers on Markov Data: Constant Depth Sufficescited by 26
  20. 2021On the Representation of Solutions to Elliptic PDEs in Barron Spacescited by 25
  21. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 23
  22. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  23. 2023Can a Transformer Represent a Kalman Filter?cited by 20
  24. 2019A Tutorial on Concentration Bounds for System Identificationcited by 19
  25. 2024How do Transformers perform In-Context Autoregressive Learning?cited by 16
  26. 2022Neural Network Approximations of PDEs Beyond Linearity: A Representational Perspectivecited by 15
  27. 2024One-Layer Transformer Provably Learns One-Nearest Neighbor In Contextcited by 14
  28. 2024Non-asymptotic Convergence of Training Transformers for Next-token Predictioncited by 13
  29. 2024On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capabilitycited by 5
  30. 2024Long-Context Linear System Identificationcited by 3
  31. 2025Transformers and Their Roles as Time Series Foundation Modelscited by 3
  32. 2024State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?cited by 2
  33. 2024Transformers as Implicit State Estimators: In-Context Learning in Dynamical Systemscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regressioncited by 0