mlsanityData to 7 Oct 2026

Linear attention is (maybe) all you need (to understand transformer optimization)

arXiv 2310.01082 ↗First posted 2 Oct 2023cs.LG cs.AI math.OCcited by 97

Not on the map. Why?

Abstract (arXiv)

Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of training Transformers by carefully studying a simple yet canonical linearized shallow Transformer model. Specifically, we train linear Transformers to solve regression tasks, inspired by J.~von Oswald et al.~(ICML 2023), and K.~Ahn et al.~(NeurIPS 2023). Most importantly, we observe that our proposed linearized models can reproduce several prominent aspects of Transformer training dynamics. Consequently, the results obtained in this paper suggest that a simple linearized Transformer model could actually be a valuable, realistic abstraction for understanding Transformer optimization.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

24 of 31 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2022Transformers learn in-context by gradient descentcited by 714
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  8. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  9. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  10. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  11. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  12. 2020Understanding the Difficulty of Training Transformerscited by 210
12 more
  1. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  2. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  3. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  4. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  5. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  6. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  7. 2022Learning threshold neurons via the "edge of stability"cited by 50
  8. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  9. 2023On the Role of Attention in Prompt-tuningcited by 49
  10. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  11. 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24
  12. 2023A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Taskscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 33; 2025: 34; 2026: 29.

97 papers.

Sort:

By area:

  1. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
  2. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  3. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  4. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  5. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  6. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  7. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 48
  8. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  9. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  10. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  11. 2024SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attentioncited by 43
  12. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
38 more
  1. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 40
  2. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  3. 2024Does SGD really happen in tiny subspaces?cited by 35
  4. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  5. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 29
  6. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  7. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 23
  8. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 22
  9. 2024Setting the Record Straight on Transformer Oversmoothingcited by 19
  10. 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
  11. 2025Can SGD Handle Heavy-Tailed Noise?cited by 15
  12. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 14
  13. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  14. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 13
  15. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  16. 2024On the Role of Depth and Looping for In-Context Learning with Task Diversitycited by 11
  17. 2024Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Taskscited by 10
  18. 2024What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysiscited by 9
  19. 2025Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transformcited by 9
  20. 2024AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methodscited by 8
  21. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  22. 2026Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decompositioncited by 8
  23. 2025On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learningcited by 7
  24. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  25. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  26. 2024Transfer Learning Beyond Bounded Density Ratioscited by 6
  27. 2024Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Datacited by 5
  28. 2024On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capabilitycited by 5
  29. 2024In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learningcited by 5
  30. 2024Learning Linear Attention in Polynomial Timecited by 5
  31. 2025Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivecited by 5
  32. 2025On the Origin of Algorithmic Progress in AIcited by 5
  33. 2026On Surprising Effectiveness of Masking Updates in Adaptive Optimizerscited by 5
  34. 2025Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimizationcited by 4
  35. 2025Understanding the Role of Training Data in Test-Time Scalingcited by 4
  36. 2026Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioningcited by 4
  37. 2024On Understanding Attention-Based In-Context Learning for Categorical Datacited by 3
  38. 2025When Can You Get Away with Low Memory Adam?cited by 3

50 most cited of 97 All 97 (JSON)