mlsanityData to 7 Oct 2026

In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learning

arXiv 2409.12293 ↗First posted 18 Sep 2024cs.LG cs.NA math.NAcited by 5

Not on the map. Why?

Abstract (arXiv)

We study theoretical guarantees for solving linear systems in-context using a linear transformer architecture. For in-domain generalization, we provide neural scaling laws that bound the generalization error in terms of the number of tasks and sizes of samples used in training and inference. For out-of-domain generalization, we find that the behavior of trained transformers under task distribution shifts depends crucially on the distribution of the tasks seen during training. We introduce a novel notion of task diversity and show that it defines a necessary and sufficient condition for pre-trained transformers generalize under task distribution shifts. We also explore applications of learning linear systems in-context, such as to in-context operator learning for PDEs. Finally, we provide some numerical experiments to validate the established theory.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

34 of 51 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2021Transformers in Vision: A Surveycited by 1,009
  5. 2022In-context Learning and Induction Headscited by 930
  6. 2022Transformers learn in-context by gradient descentcited by 714
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  10. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  12. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
22 more
  1. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  2. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  3. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
  4. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  5. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  6. 2023In-Context Convergence of Transformerscited by 112
  7. 2023In-Context Operator Learning with Data Prompts for Differential Equation Problemscited by 98
  8. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  9. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  10. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  11. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  12. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 71
  13. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  14. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 46
  15. 2024Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolationcited by 45
  16. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  17. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
  18. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
  19. 2024PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Lawscited by 30
  20. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 30
  21. 2023Can a Transformer Represent a Kalman Filter?cited by 20
  22. 2024How do Transformers perform In-Context Autoregressive Learning?cited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 4.

5 papers.

  1. 2024From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiencycited by 28
  2. 2026Chain of Operators: An Inference-Time Harness for In-Context Operator Learningcited by 2
  3. 2026A Theory of Diversity for Random Matrices with Applications to In-Context Learning of Schr"odinger Equationscited by 0
  4. 2026Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learningcited by 0
  5. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0