mlsanityData to 7 Oct 2026

Going Beyond Linear Transformers with Recurrent Fast Weight Programmers

arXiv 2106.06295 ↗First posted 11 Jun 2021cs.LGcited by 84

Not on the map. Why?

Abstract (arXiv)

Transformers with linearised attention (''linear Transformers'') have demonstrated the practical scalability and effectiveness of outer product-based Fast Weight Programmers (FWPs) from the '90s. However, the original FWP formulation is more general than the one of linear Transformers: a slow neural network (NN) continually reprograms the weights of a fast NN with arbitrary architecture. In existing linear Transformers, both NNs are feedforward and consist of a single layer. Here we explore new variations by adding recurrence to the slow and fast nets. We evaluate our novel recurrent FWPs (RFWPs) on two synthetic algorithmic tasks (code execution and sequential ListOps), Wikitext-103 language models, and on the Atari 2600 2D game environment. Our models exhibit properties of Transformers and RNNs. In the reinforcement learning setting, we report large improvements over LSTM in several Atari games. Our code is public.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 57 of its 84 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

57 of 84 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2016Layer Normalizationcited by 5,579
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
  1. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  7. 2016HyperNetworkscited by 1,003
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  11. 2014Neural Turing Machinescited by 630
  12. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  13. 2021Random Feature Attentioncited by 350
  14. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  15. 2016Dynamic Filter Networkscited by 318
  16. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  17. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  18. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  19. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  20. 2015Highway Networkscited by 257
  21. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  22. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  23. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  24. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  25. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  26. 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
  27. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  28. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  29. 2014Learning to Executecited by 105
  30. 2019Language Modeling with Deep Transformerscited by 96
  31. 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 83
  32. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  33. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80
  34. 2020Learning Associative Inference Using Fast Weight Memorycited by 51
  35. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  36. 2019Metalearned Neural Memorycited by 44
  37. 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
  38. 2019TorchBeast: A PyTorch Platform for Distributed RLcited by 41

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 15; 2023: 14; 2024: 15; 2025: 18; 2026: 20.

84 papers.

Sort:

By area:

  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  3. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  4. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  5. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  6. 2024Titans: Learning to Memorize at Test Timecited by 303
  7. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  8. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  9. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  10. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  11. 2022Block-Recurrent Transformerscited by 107
  12. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
38 more
  1. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  2. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  3. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  4. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  5. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  6. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  7. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  8. 2025End-to-End Test-Time Training for Long Contextcited by 53
  9. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  10. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
  11. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 40
  12. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  13. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 24
  14. 2023Efficient Attention via Control Variatescited by 23
  15. 2022Meta-Learning Fast Weight Language Modelscited by 20
  16. 2023Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensionscited by 20
  17. 2023SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attentioncited by 20
  18. 2023Exploring the Promise and Limits of Real-Time Recurrent Learningcited by 19
  19. 2025Lattice: Learning to Efficiently Compress the Memorycited by 18
  20. 2023Trainable Transformer in Transformercited by 17
  21. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 17
  22. 2022Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rulescited by 15
  23. 2022Unsupervised Learning of Temporal Abstractions with Slot-based Transformerscited by 14
  24. 2023Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selectioncited by 14
  25. 2022Learning to Generalize with Object-centric Agents in the Open World Survival Game Craftercited by 13
  26. 2024Learning Useful Representations of Recurrent Neural Network Weight Matricescited by 13
  27. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  28. 2023Learning to (Learn at Test Time)cited by 12
  29. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  30. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 12
  31. 2026Memory Caching: RNNs with Growing Memorycited by 12
  32. 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learningcited by 10
  33. 2022Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rulescited by 9
  34. 2024Rethinking Transformers in Solving POMDPscited by 9
  35. 2025Sequential-Parallel Duality in Prefix Scannable Modelscited by 8
  36. 2025TNT: Improving Chunkwise Training for Test-Time Memorizationcited by 7
  37. 2023Metalearning Continual Learning Algorithmscited by 6
  38. 2024Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyphcited by 6

50 most cited of 84 All 84 (JSON)