mlsanityData to 7 Oct 2026

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness

arXiv 2402.11639 ↗First posted 18 Feb 2024cs.LG cs.AI cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

A striking property of transformers is their ability to perform in-context learning (ICL), a machine learning framework in which the learner is presented with a novel context during inference implicitly through some data, and tasked with making a prediction in that context. As such, that learner must adapt to the context without additional training. We explore the role of softmax attention in an ICL setting where each context encodes a regression task. We show that an attention unit learns a window that it uses to implement a nearest-neighbors predictor adapted to the landscape of the pretraining tasks. Specifically, we show that this window widens with decreasing Lipschitzness and increasing label noise in the pretraining tasks. We also show that on low-rank, linear problems, the attention unit learns to project onto the appropriate subspace before inference. Further, we show that this adaptivity relies crucially on the softmax activation and thus cannot be replicated by the linear activation often studied in prior theoretical analyses.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 35 of its 62 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

35 of 62 references.

References · 35

  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2022In-context Learning and Induction Headscited by 930
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  6. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  7. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  8. 2022Transformers Learn Shortcuts to Automatacited by 260
  9. 2023Looped Transformers as Programmable Computerscited by 224
  10. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  11. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  12. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
23 more
  1. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  2. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  3. 2023Representational Strengths and Limitations of Transformerscited by 132
  4. 2023In-Context Convergence of Transformerscited by 112
  5. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  6. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  7. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  8. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  9. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  10. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  11. 2023Transformers as Support Vector Machinescited by 67
  12. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  13. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  14. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  15. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  16. 2021On the Expressive Power of Self-Attention Matricescited by 44
  17. 2023What can a Single Attention Layer Learn? A Study Through the Random Features Lenscited by 39
  18. 2023Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representationcited by 28
  19. 2023Understanding Emergent In-Context Learning from a Kernel Regression Perspectivecited by 23
  20. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  21. 2023Why Softmax Attention Outperforms Linear Attentioncited by 15
  22. 2020On the Ability and Limitations of Transformers to Recognize Formal Languagescited by 12
  23. 2022Designing Robust Transformers using Robust Kernel Density Estimationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 11; 2026: 3.

22 papers.

Sort:

By area:

  1. 2024Mechanics of Next Token Prediction with Self-Attentioncited by 34
  2. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  3. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 22
  4. 2024Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyondcited by 15
  5. 2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 11
  6. 2024Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selectioncited by 9
  7. 2025From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamicscited by 9
  8. 2025Transformer learns the cross-task prior and regularization for in-context learningcited by 7
  9. 2024Context-Scaling versus Task-Scaling in In-Context Learningcited by 6
  10. 2025On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspectivecited by 6
  11. 2024On the Power of Convolution Augmented Transformercited by 5
  12. 2025When and How Unlabeled Data Provably Improve In-Context Learningcited by 5
10 more
  1. 2025Provable In-Context Learning of Nonlinear Regression with Transformerscited by 4
  2. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
  3. 2025StaICC: Standardized Evaluation for Classification Task in In-context Learningcited by 2
  4. 2025Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyondcited by 2
  5. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  6. 2025Softmax ≥ Linear: Transformers may learn to classify in-context by kernel gradient descentcited by 2
  7. 2026Affine-Scaled Attention: Towards Flexible and Stable Transformer Attentioncited by 2
  8. 2025In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separationcited by 1
  9. 2026Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regressioncited by 0
  10. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0