mlsanityData to 14 Oct 2026

Softmax ≥ Linear: Transformers may learn to classify in-context by kernel gradient descent

arXiv 2510.10425 ↗First posted 12 Oct 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The remarkable ability of transformers to learn new concepts solely by reading examples within the input prompt, termed in-context learning (ICL), is a crucial aspect of intelligent behavior. Here, we focus on understanding the learning algorithm transformers use to learn from context. Existing theoretical work, often based on simplifying assumptions, has primarily focused on linear self-attention and continuous regression tasks, finding transformers can learn in-context by gradient descent. Given that transformers are typically trained on discrete and complex tasks, we bridge the gap from this existing work to the setting of classification, with non-linear (importantly, softmax) activation. We find that transformers still learn to do gradient descent in-context, though on functionals in the kernel feature space and with a context-adaptive learning rate in the case of softmax transformer. These theoretical findings suggest a greater adaptability to context for softmax attention, which we empirically verify and study through ablations. Overall, we hope this enhances theoretical understanding of in-context learning algorithms in more realistic settings, pushes forward our intuitions and enables further theory bridging to larger models.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 45 of its 48 references that are in the map. First posted on arXiv 12 Oct 2025.

Builds on

45 of 48 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2016Layer Normalizationcited by 5,669
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  7. 2022In-context Learning and Induction Headscited by 964
  8. 2022Transformers learn in-context by gradient descentcited by 732
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  11. 2016Learning to learn by gradient descent by gradient descentcited by 620
  12. 2023Larger language models do in-context learning differentlycited by 394
33 more
  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  2. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  3. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  4. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  5. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  6. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  7. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  8. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
  9. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 118
  10. 2023In-Context Convergence of Transformerscited by 115
  11. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  12. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 89
  13. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
  14. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 74
  15. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  16. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  17. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  18. 2024Asymptotic theory of in-context learning by linear attentioncited by 55
  19. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 54
  20. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  21. 2024Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscited by 40
  22. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  23. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  24. 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 33
  25. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  26. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  27. 2024Toward Understanding In-context vs. In-weight Learningcited by 19
  28. 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
  29. 2025Strategy Coopetition Explains the Emergence and Transience of In-Context Learningcited by 17
  30. 2024One-Layer Transformer Provably Learns One-Nearest Neighbor In Contextcited by 16
  31. 2024Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgettingcited by 15
  32. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 14
  33. 2024On Understanding Attention-Based In-Context Learning for Categorical Datacited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspectivecited by 3
  2. 2026Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regressioncited by 1
  3. 2026In-context Learning of Single-index Targets: Comparing Kernel and Feature Learnerscited by 1
  4. 2026How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learningcited by 0