mlsanityData to 7 Oct 2026

Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context

arXiv 2312.06528 ↗First posted 11 Dec 2023cs.LGcited by 63

Not on the map. Why?

Abstract (arXiv)

Many neural network architectures are known to be Turing Complete, and can thus, in principle implement arbitrary algorithms. However, Transformers are unique in that they can implement gradient-based learning algorithms under simple parameter configurations. This paper provides theoretical and empirical evidence that (non-linear) Transformers naturally learn to implement gradient descent in function space, which in turn enable them to learn non-linear functions in context. Our results apply to a broad class of combinations of non-linear architectures and non-linear in-context learning tasks. Additionally, we show that the optimal choice of non-linear activation depends in a natural way on the class of functions that need to be learned.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

27 of 45 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2022In-context Learning and Induction Headscited by 930
  5. 2022Transformers learn in-context by gradient descentcited by 714
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  9. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  10. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  11. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  12. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
15 more
  1. 2023Looped Transformers as Programmable Computerscited by 224
  2. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  3. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  4. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  5. 2023In-Context Convergence of Transformerscited by 112
  6. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  7. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  8. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  9. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  10. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  11. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  12. 2023Replacing softmax with ReLU in Vision Transformerscited by 43
  13. 2021Improving Transformers with Probabilistic Attention Keyscited by 28
  14. 2023Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representationcited by 28
  15. 2021Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machinescited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 23; 2026: 12.

63 papers.

Sort:

By area:

  1. 2024Can large language models explore in-context?cited by 75
  2. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  3. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  4. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  5. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  6. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  7. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  8. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 23
  9. 2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
  10. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21
  11. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  12. 2024On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixturescited by 15
38 more
  1. 2024Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysiscited by 14
  2. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 13
  3. 2024Non-asymptotic Convergence of Training Transformers for Next-token Predictioncited by 13
  4. 2024A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstrationcited by 13
  5. 2025Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methodscited by 13
  6. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  7. 2024Abrupt Learning in Transformers: A Case Study on Matrix Completioncited by 11
  8. 2025How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Biascited by 10
  9. 2024Towards Understanding the Universality of Transformers for Next-Token Predictioncited by 9
  10. 2024Transformers Handle Endogeneity in In-Context Linear Regressioncited by 8
  11. 2025On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learningcited by 7
  12. 2024Context-Scaling versus Task-Scaling in In-Context Learningcited by 6
  13. 2024On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recoverycited by 6
  14. 2024Task Diversity Shortens the ICL Plateaucited by 5
  15. 2024Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Modelscited by 4
  16. 2025On the Robustness of Transformers against Context Hijacking for Linear Classificationcited by 4
  17. 2025Transformers Meet In-Context Learning: A Universal Approximation Theorycited by 4
  18. 2025Provable In-Context Learning of Nonlinear Regression with Transformerscited by 4
  19. 2024On Understanding Attention-Based In-Context Learning for Categorical Datacited by 3
  20. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  21. 2025Disentangling Feature Structure: A Mathematically Provable Two-Stage Training Dynamics in Transformerscited by 3
  22. 2025Continuum Transformers Perform In-Context Learning by Operator Gradient Descentcited by 3
  23. 2025Test time training enhances in-context learning of nonlinear functionscited by 3
  24. 2025Transmuting prompts into weightscited by 3
  25. 2025The Initialization Determines Whether In-Context Learning Is Gradient Descentcited by 3
  26. 2024No Free Lunch for Defending Against Prefilling Attack by In-Context Learningcited by 2
  27. 2025Adversarially Pretrained Transformers May Be Universally Robust In-Context Learnerscited by 2
  28. 2025Softmax ≥ Linear: Transformers may learn to classify in-context by kernel gradient descentcited by 2
  29. 2026Unlabeled Data Can Provably Enhance In-Context Learning of Transformerscited by 2
  30. 2026Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizercited by 2
  31. 2026Large language models reorganize representational geometry during in-context learningcited by 2
  32. 2024Transformers as Game Players: Provable In-context Game-playing Capabilities of Pre-trained Modelscited by 1
  33. 2025How Transformers Learn In-Context Recall Tasks? Optimality, Training Dynamics and Generalizationcited by 1
  34. 2025Assessing the Limits of In-Context Learning beyond Functions using Partially Ordered Relationcited by 1
  35. 2025On the Existence of Universal Simulators of Attentioncited by 1
  36. 2025Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learningcited by 1
  37. 2025Optimal In-context Adaptivity and Distributional Robustness of Transformerscited by 1
  38. 2025In-Context Semi-Supervised Learningcited by 1

50 most cited of 63 All 63 (JSON)