mlsanityData to 7 Oct 2026

Using Fast Weights to Attend to the Recent Past

arXiv 1610.06258 ↗First posted 20 Oct 2016stat.ML cs.LG cs.NEcited by 161

Not on the map. Why?

Abstract (arXiv)

Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower than activities but much faster than the standard weights. These "fast weights" can be used to store temporary memories of the recent past and they provide a neurally plausible way of implementing the type of attention to the past that has recently proved very helpful in sequence-to-sequence models. By using fast weights we can avoid the need to store copies of neural activity patterns.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Oct 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 23; 2023: 15; 2024: 22; 2025: 31; 2026: 53.

161 papers.

Sort:

By area:

  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  3. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  4. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  5. 2021Random Feature Attentioncited by 350
  6. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  7. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  8. 2022Recurrent Memory Transformercited by 197
  9. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  10. 2021How to represent part-whole hierarchies in a neural networkcited by 123
  11. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  12. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
38 more
  1. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  2. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  3. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  4. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  5. 2021Relating transformers to models and neural representations of the hippocampal formationcited by 62
  6. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 62
  7. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
  8. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
  9. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  10. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 40
  11. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  12. 2024Separations in the Representational Capabilities of Transformers and Recurrent Architecturescited by 29
  13. 2022Fine-Tuning Pre-trained Transformers into Decaying Fast Weightscited by 26
  14. 2024Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Passcited by 26
  15. 2025Latent Thought Models with Variational Bayes Inference-Time Computationcited by 25
  16. 2023Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutionscited by 24
  17. 2025Key-value memory in the braincited by 22
  18. 2022Open Long-Tailed Recognition in a Dynamic Worldcited by 21
  19. 2021Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolutioncited by 20
  20. 2022Meta-Learning Fast Weight Language Modelscited by 20
  21. 2023Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensionscited by 20
  22. 2021Hard-Attention for Scalable Image Classificationcited by 18
  23. 2023Trainable Transformer in Transformercited by 17
  24. 2023Traveling Waves Encode the Recent Past and Enhance Sequence Learningcited by 16
  25. 2023Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Abilitycited by 16
  26. 2024Self-Contrastive Forward-Forward Algorithmcited by 15
  27. 2026In-Place Test-Time Trainingcited by 15
  28. 2025Lightweight Online Adaption for Time Series Foundation Model Forecastscited by 14
  29. 2022Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learningcited by 13
  30. 2024Towards LifeSpan Cognitive Systemscited by 13
  31. 2023Learning to (Learn at Test Time)cited by 12
  32. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  33. 2021Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memoriescited by 11
  34. 2021Adaptive Convolutions with Per-pixel Dynamic Filter Atomcited by 11
  35. 2026Fast-weight Product Key Memorycited by 11
  36. 2022A Call for Clarity in Beam Search: How It Works and When It Stopscited by 10
  37. 2023PTGB: Pre-Train Graph Neural Networks for Brain Network Analysiscited by 10
  38. 2022Characterizing Verbatim Short-Term Memory in Neural Language Modelscited by 8

50 most cited of 161 All 161 (JSON)