Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks
Not on the map. Why?
Abstract (arXiv)
Short-term memory in standard, general-purpose, sequence-processing recurrent neural networks (RNNs) is stored as activations of nodes or "neurons." Generalising feedforward NNs to such RNNs is mathematically straightforward and natural, and even historical: already in 1943, McCulloch and Pitts proposed this as a surrogate to "synaptic modifications" (in effect, generalising the Lenz-Ising model, the first non-sequence processing RNN architecture of the 1920s). A lesser known alternative approach to storing short-term memory in "synaptic connections" -- by parameterising and controlling the dynamics of a context-sensitive time-varying weight matrix through another NN -- yields another "natural" type of short-term memory in sequence processing NNs: the Fast Weight Programmers (FWPs) of the early 1990s. FWPs have seen a recent revival as generic sequence processors, achieving competitive performance across various tasks. They are formally closely related to the now popular Transformers. Here we present them in the context of artificial NNs as an abstraction of biological NNs -- a perspective that has not been stressed enough in previous FWP work. We first review aspects of FWPs for pedagogical purposes, then discuss connections to related works motivated by insights from neuroscience.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 64 references that are in the map. First posted on arXiv 17 Nov 2022.
Builds on
26 of 64 references.
References · 26
- 2017Attention Is All You Needcited by 59,932
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2020Rethinking Attention with Performerscited by 1,584
- 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
- 2020Hopfield Networks is All You Needcited by 461
- 2017Meta Networkscited by 268
- 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 110
- 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
- 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 83
- 2021Relating transformers to models and neural representations of the hippocampal formationcited by 62
- 2022Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Modelscited by 52
- 2020Learning Associative Inference Using Fast Weight Memorycited by 51
14 more
- 2020Meta Learning Backpropagation And Improving Itcited by 49
- 2020Meta-Learning through Hebbian Plasticity in Random Networkscited by 47
- 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
- 2019Metalearned Neural Memorycited by 44
- 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 40
- 2020Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticitycited by 39
- 2018Learning to Reason with Third-Order Tensor Productscited by 27
- 2018Metalearning with Hebbian Fast Weightscited by 15
- 2022Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rulescited by 15
- 2022Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rulescited by 9
- 2020Enabling Continual Learning with Differentiable Hebbian Plasticitycited by 8
- 2020Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modellingcited by 5
- 2022Short-Term Plasticity Neurons Learning to Learn and Forgetcited by 5
- 2022The molecular memory code and synaptic plasticity: a synthesiscited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.