mlsanityData to 14 Oct 2026

Fast weight programming and linear transformers: from machine learning to neurobiology

arXiv 2508.08435 ↗First posted 11 Aug 2025cs.LG cs.AI q-bio.NCcited by 8

Not on the map. Why?

Abstract (arXiv)

Recent advances in artificial neural networks for machine learning, and language modeling in particular, have established a family of recurrent neural network (RNN) architectures that, unlike conventional RNNs with vector-form hidden states, use two-dimensional (2D) matrix-form hidden states. Such 2D-state RNNs, known as Fast Weight Programmers (FWPs), can be interpreted as a neural network whose synaptic weights (called fast weights) dynamically change over time as a function of input observations, and serve as short-term memory storage; corresponding synaptic weight modifications are controlled or programmed by another network (the programmer) whose parameters are trained (e.g., by gradient descent). In this Primer, we review the technical foundations of FWPs, their computational characteristics, and their connections to transformers and state space models. We also discuss connections between FWPs and models of synaptic plasticity in the brain, suggesting a convergence of natural and artificial intelligence.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 118 of its 242 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

118 of 242 references.

References · 118

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2016Layer Normalizationcited by 5,669
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
38 more
  1. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  3. 2020Rethinking Attention with Performerscited by 1,601
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  5. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  6. 2016HyperNetworkscited by 1,027
  7. 2014Deep Learning in Neural Networks: An Overviewcited by 770
  8. 2022Transformers learn in-context by gradient descentcited by 732
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  10. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  11. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  12. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  13. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  14. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  15. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  16. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  17. 2020Hopfield Networks is All You Needcited by 475
  18. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  19. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  20. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  21. 2016Learning to reinforcement learncited by 377
  22. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  23. 2021Random Feature Attentioncited by 354
  24. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  25. 2024Titans: Learning to Memorize at Test Timecited by 328
  26. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  27. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  28. 2015LSTM: A Search Space Odysseycited by 316
  29. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  30. 2017Meta Networkscited by 269
  31. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
  32. 2015Highway Networkscited by 260
  33. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  34. 2020On the Binding Problem in Artificial Neural Networkscited by 233
  35. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  36. 2022Neural Networks and the Chomsky Hierarchycited by 217
  37. 2016A Decomposable Attention Model for Natural Language Inferencecited by 215
  38. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193

50 most cited of 118 All 118 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
  2. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
  3. 2026In-Place Test-Time Trainingcited by 20
  4. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 8
  5. 2026Learning a Vector-Symbolic Model for Socio-Cultural Taskscited by 0
  6. 2026Path-dependent Discrete Amortized Inferencecited by 0
  7. 2026Switching Linear Attentioncited by 0
  8. 2026Continuous Memory Machinescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A brain basis of dynamical intelligence for AI and computational neuroscience
  2. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the story
  3. 2024Structurally Flexible Neural Networks: Evolving the Building Blocks for General Agents
  4. 2025The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain
  5. 2025The impact of allocation strategies in subset learning on the expressive power of neural networks
  6. 2024Deep activity propagation via weight initialization in spiking neural networks
  7. 2025Generalizable, real-time neural decoding with hybrid state-space models
  8. 2025Scaling Up Resonate-and-Fire Networks for Fast Deep Learning