mlsanityData to 7 Oct 2026

Linear Transformers are Versatile In-Context Learners

arXiv 2402.14180 ↗First posted 21 Feb 2024cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Recent research has demonstrated that transformers, particularly linear attention models, implicitly execute gradient-descent-like algorithms on data provided in-context during their forward inference step. However, their capability in handling more complex problems remains unexplored. In this paper, we prove that each layer of a linear transformer maintains a weight vector for an implicit linear regression problem and can be interpreted as performing a variant of preconditioned gradient descent. We also investigate the use of linear transformers in a challenging scenario where the training data is corrupted with different levels of noise. Remarkably, we demonstrate that for this problem linear transformers discover an intricate and highly effective optimization algorithm, surpassing or matching in performance many reasonable baselines. We analyze this algorithm and show that it is a novel approach incorporating momentum and adaptive rescaling based on noise levels. Our findings show that even linear transformers possess the surprising ability to discover sophisticated optimization strategies.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 35 of its 42 references that are in the map. First posted on arXiv 21 Feb 2024.

Builds on

35 of 42 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Mistral 7Bcited by 3,065
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2022Transformers learn in-context by gradient descentcited by 714
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
23 more
  1. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  2. 2023Larger language models do in-context learning differentlycited by 386
  3. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  4. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  5. 2023In-Context Learning Creates Task Vectorscited by 302
  6. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  7. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  8. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  9. 2023Looped Transformers as Programmable Computerscited by 224
  10. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  11. 2023Large Language Models as General Pattern Machinescited by 196
  12. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  13. 2023In-Context Convergence of Transformerscited by 112
  14. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  15. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  16. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  17. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  18. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  19. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  20. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  21. 2023Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscited by 45
  22. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  23. 2023Transformers can optimally learn regression mixture modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 13; 2026: 8.

28 papers.

Sort:

By area:

  1. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  2. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  3. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  4. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 13
  5. 2024Ask, and it shall be given: On the Turing completeness of promptingcited by 10
  6. 2025Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2Bcited by 10
  7. 2024Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lenscited by 7
  8. 2025Transformer learns the cross-task prior and regularization for in-context learningcited by 7
  9. 2024On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capabilitycited by 5
  10. 2025Transformers Meet In-Context Learning: A Universal Approximation Theorycited by 4
  11. 2025Learning Causal Graphs at Scale: A Foundation Model Approachcited by 4
  12. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
16 more
  1. 2025Softmax ≥ Linear: Transformers may learn to classify in-context by kernel gradient descentcited by 2
  2. 2025In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separationcited by 1
  3. 2025Neural Interpretable PDEs: Harmonizing Fourier Insights with Attention for Scalable and Interpretable Physics Discoverycited by 1
  4. 2025Multi-Layer Attention is the Amplifier of Demonstration Effectivenesscited by 1
  5. 2025Linear Transformers Implicitly Discover Unified Numerical Algorithmscited by 1
  6. 2026Bayesian Optimality of In-Context Learning with Selective State Spacescited by 1
  7. 2024Toward generalizable learning of all (linear) first-order methods via memory augmented Transformerscited by 0
  8. 2025How Private is Your Attention? Bridging Privacy with In-Context Learningcited by 0
  9. 2025From Text to Trajectories: GPT-2 as an ODE Solver via In-Contextcited by 0
  10. 2026Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMscited by 0
  11. 2026Ordinary Least Squares is a Special Case of Transformercited by 0
  12. 2026Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regressioncited by 0
  13. 2026In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspectivecited by 0
  14. 2026Transformers Can Learn Posterior Predictive Distributions In-Contextcited by 0
  15. 2026Neural Events: Discrete Asynchronous Autoencoders for Event-Based Visioncited by 0
  16. 2026Dendritic In-Context Learning in a Single-Layer Spiking Neural Networkcited by 0