mlsanityData to 13 Oct 2026

Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond

arXiv 2407.10005 ↗First posted 13 Jul 2024cs.LG cs.AI cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Recent research has shown that Transformers with linear attention are capable of in-context learning (ICL) by implementing a linear estimator through gradient descent steps. However, the existing results on the optimization landscape apply under stylized settings where task and feature vectors are assumed to be IID and the attention weights are fully parameterized. In this work, we develop a stronger characterization of the optimization and generalization landscape of ICL through contributions on architectures, low-rank parameterization, and correlated designs: (1) We study the landscape of 1-layer linear attention and 1-layer H3, a state-space model. Under a suitable correlated design assumption, we prove that both implement 1-step preconditioned gradient descent. We show that thanks to its native convolution filters, H3 also has the advantage of implementing sample weighting and outperforming linear attention in suitable settings. (2) By studying correlated designs, we provide new risk bounds for retrieval augmented generation (RAG) and task-feature alignment which reveal how ICL sample complexity benefits from distributional alignment. (3) We derive the optimal risk for low-rank parameterized attention weights in terms of covariance spectrum. Through this, we also shed light on how LoRA can adapt to a new distribution by capturing the shift between task covariances. Experimental results corroborate our theoretical findings. Overall, this work explores the optimization and risk landscape of ICL in practically meaningful settings and contributes to a more thorough understanding of its mechanics.

Key facts. Cited by 24 papers in the map (data through 13 Oct 2026). Builds on 47 of its 61 references that are in the map. First posted on arXiv 13 Jul 2024.

Builds on

47 of 61 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,658
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,440
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,229
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,994
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,736
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
35 more
  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 947
  2. 2022In-context Learning and Induction Headscited by 940
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 925
  4. 2022Transformers learn in-context by gradient descentcited by 720
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 706
  6. 2016Language Modeling with Gated Convolutional Networkscited by 694
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 676
  8. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 652
  9. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 565
  11. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  12. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 483
  13. 2021Transformers Can Do Bayesian Inferencecited by 346
  14. 2023Trained Transformers Learn Linear Models In-Contextcited by 333
  15. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 293
  16. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 276
  17. 2022Transformers Learn Shortcuts to Automatacited by 261
  18. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 221
  19. 2024Many-Shot In-Context Learningcited by 205
  20. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 174
  21. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 169
  22. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  23. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  24. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  25. 2023Uncovering mesa-optimization algorithms in Transformerscited by 97
  26. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  27. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 70
  28. 2022Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Datacited by 61
  29. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  30. 2024Is Mamba Capable of In-Context Learning?cited by 45
  31. 2024Dual Operating Modes of In-Context Learningcited by 43
  32. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  33. 2023Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Abilitycited by 16
  34. 2023Can Transformers Learn Optimal Filtering for Unknown Systems?cited by 15
  35. 2023Gated recurrent neural networks discover attentioncited by 7

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 8.

24 papers.

Sort:

By area:

  1. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 45
  2. 2025Test-Time Training Provably Improves Transformers as In-context Learnerscited by 16
  3. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 12
  4. 2024On the Role of Depth and Looping for In-Context Learning with Task Diversitycited by 11
  5. 2025Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivecited by 6
  6. 2025Pretrain-Test Task Alignment Governs Generalization in In-Context Learningcited by 6
  7. 2025When and How Unlabeled Data Provably Improve In-Context Learningcited by 5
  8. 2025On the Robustness of Transformers against Context Hijacking for Linear Classificationcited by 4
  9. 2025Provable Benefits of Task-Specific Prompts for In-context Learningcited by 4
  10. 2025In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Flycited by 4
  11. 2025Asymptotic Study of In-context Learning with Random Transformers through Equivalent Modelscited by 3
  12. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
12 more
  1. 2026Learning to Adapt: In-Context Learning Beyond Stationaritycited by 2
  2. 2025Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learningcited by 1
  3. 2025How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPscited by 1
  4. 2025Optimal Attention Temperature Improves the Robustness of In-Context Learning under Distribution Shift in High Dimensionscited by 1
  5. 2026Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Modelscited by 1
  6. 2025Optimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functionscited by 0
  7. 2026Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMscited by 0
  8. 2026Covariance-Aware Transformers for Quadratic Programming and Decision Makingcited by 0
  9. 2026The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspacescited by 0
  10. 2026Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatchcited by 0
  11. 2026Generalization of Transformer-Based Neural Quantum States via In-Context Learningcited by 0
  12. 2026Retrieval-Based In-Context Learning: A Domain Adaptation Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Probing In-Context Learning: Impact of Task Complexity and Model Architecture on Generalization and Efficiency
  2. 2025In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention
  3. 2025How Transformers Learn In-Context Recall Tasks? Optimality, Training Dynamics and Generalization
  4. 2026Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning
  5. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learning
  6. 2026Induction Heads Interpolate N-Grams
  7. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
  8. 2026A Comparative Analysis of Attention versus State-Space Models for In-Context Learning