mlsanityData to 7 Oct 2026

Linear Transformers Are Secretly Fast Weight Programmers

arXiv 2102.11174 ↗First posted 22 Feb 2021cs.LGcited by 471

Not on the map. Why?

Cited by 26 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow" neural net learns by gradient descent to program the ``fast weights" of another net through sequences of elementary programming instructions which are additive outer products of self-invented activation patterns (today called keys and values). Such Fast Weight Programmers (FWPs) learn to manipulate the contents of a finite memory and dynamically interact with it. We infer a memory capacity limitation of recent linearised softmax attention variants, and replace the purely additive outer products by a delta rule-like programming instruction, such that the FWP can more easily learn to correct the current mapping from keys to values. The FWP also learns to compute dynamically changing learning rates. We also propose a new kernel function to linearise attention which balances simplicity and effectiveness. We conduct experiments on synthetic retrieval problems as well as standard machine translation and language modelling tasks which demonstrate the benefits of our methods.

Key facts. Cited by 471 papers in the map (data through 7 Oct 2026). Builds on 41 of its 72 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

41 of 72 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
29 more
  1. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  2. 2016HyperNetworkscited by 1,003
  3. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  4. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  5. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  6. 2020Hopfield Networks is All You Needcited by 461
  7. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  8. 2016Dynamic Filter Networkscited by 318
  9. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  10. 2017Meta Networkscited by 268
  11. 2016Dense Associative Memory for Pattern Recognitioncited by 230
  12. 2020On the Binding Problem in Artificial Neural Networkscited by 230
  13. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  14. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  15. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  16. 2018Scaling Neural Machine Translationcited by 204
  17. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  18. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  19. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  20. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  21. 2019Language Modeling with Deep Transformerscited by 96
  22. 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 83
  23. 2020Learning Associative Inference Using Fast Weight Memorycited by 51
  24. 2020Meta Learning Backpropagation And Improving Itcited by 49
  25. 2019Metalearned Neural Memorycited by 44
  26. 2020Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticitycited by 39
  27. 2018Learning to Reason with Third-Order Tensor Productscited by 27
  28. 2015Image Question Answering using Convolutional Neural Network with Dynamic Parameter Predictioncited by 26
  29. 2018Metalearning with Hebbian Fast Weightscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 34; 2023: 43; 2024: 70; 2025: 103; 2026: 209.

471 papers.

Sort:

By area:

  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  3. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  4. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  5. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  6. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  7. 2021Choose a Transformer: Fourier or Galerkincited by 333
  8. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  9. 2024Titans: Learning to Memorize at Test Timecited by 303
  10. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  11. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  12. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
38 more
  1. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  2. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  3. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  4. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  5. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  6. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  7. 2025Test-Time Training Done Rightcited by 121
  8. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  9. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  10. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  11. 2022Block-Recurrent Transformerscited by 107
  12. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  13. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  14. 2026Kimi K3: Open Frontier Intelligencecited by 99
  15. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  16. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  17. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  18. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  19. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  20. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  21. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  22. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  23. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  24. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  25. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  26. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  27. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
  28. 2025Long-Context State-Space Video World Modelscited by 59
  29. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  30. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 57
  31. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  32. 2026Attention Residualscited by 57
  33. 2024Faster Diffusion via Temporal Attention Decompositioncited by 54
  34. 2025End-to-End Test-Time Training for Long Contextcited by 53
  35. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  36. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
  37. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  38. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44

50 most cited of 471 All 471 (JSON)