# Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks

arXiv: https://arxiv.org/abs/2211.09440 · HTML: https://mlsanity.com/paper/2211.09440/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 64 references that are in the map.
- First posted on arXiv 17 Nov 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [Large Associative Memory Problem in Neurobiology and Machine Learning](https://mlsanity.com/paper/2008.06996/) (2020)
- [Going Beyond Linear Transformers with Recurrent Fast Weight Programmers](https://mlsanity.com/paper/2106.06295/) (2021)
- [Differentiable plasticity: training plastic neural networks with backpropagation](https://mlsanity.com/paper/1804.02464/) (2018)
- [Relating transformers to models and neural representations of the hippocampal formation](https://mlsanity.com/paper/2112.04035/) (2021)
- [Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models](https://mlsanity.com/paper/2202.04557/) (2022)
- [Learning Associative Inference Using Fast Weight Memory](https://mlsanity.com/paper/2011.07831/) (2020)
- [Meta Learning Backpropagation And Improving It](https://mlsanity.com/paper/2012.14905/) (2020)
- [Meta-Learning through Hebbian Plasticity in Random Networks](https://mlsanity.com/paper/2007.02686/) (2020)
- [The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention](https://mlsanity.com/paper/2202.05798/) (2022)
- [Metalearned Neural Memory](https://mlsanity.com/paper/1907.09720/) (2019)
- [Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity](https://mlsanity.com/paper/2002.10585/) (2020)
- [A Modern Self-Referential Weight Matrix That Learns to Modify Itself](https://mlsanity.com/paper/2202.05780/) (2022)
- [Learning to Reason with Third-Order Tensor Products](https://mlsanity.com/paper/1811.12143/) (2018)
- [Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules](https://mlsanity.com/paper/2206.01649/) (2022)
- [Metalearning with Hebbian Fast Weights](https://mlsanity.com/paper/1807.05076/) (2018)
- [Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules](https://mlsanity.com/paper/2210.06184/) (2022)
- [Enabling Continual Learning with Differentiable Hebbian Plasticity](https://mlsanity.com/paper/2006.16558/) (2020)
- [Short-Term Plasticity Neurons Learning to Learn and Forget](https://mlsanity.com/paper/2206.14048/) (2022)
- [Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling](https://mlsanity.com/paper/2009.01803/) (2020)
- [The molecular memory code and synaptic plasticity: a synthesis](https://mlsanity.com/paper/2209.04923/) (2022)

## Built on by (most cited first, 1 of 1)

- [Fast weight programming and linear transformers: from machine learning to neurobiology](https://mlsanity.com/paper/2508.08435/) (2025)

## Abstract (arXiv)

Short-term memory in standard, general-purpose, sequence-processing recurrent neural networks (RNNs) is stored as activations of nodes or "neurons." Generalising feedforward NNs to such RNNs is mathematically straightforward and natural, and even historical: already in 1943, McCulloch and Pitts proposed this as a surrogate to "synaptic modifications" (in effect, generalising the Lenz-Ising model, the first non-sequence processing RNN architecture of the 1920s). A lesser known alternative approach to storing short-term memory in "synaptic connections" -- by parameterising and controlling the dynamics of a context-sensitive time-varying weight matrix through another NN -- yields another "natural" type of short-term memory in sequence processing NNs: the Fast Weight Programmers (FWPs) of the early 1990s. FWPs have seen a recent revival as generic sequence processors, achieving competitive performance across various tasks. They are formally closely related to the now popular Transformers. Here we present them in the context of artificial NNs as an abstraction of biological NNs -- a perspective that has not been stressed enough in previous FWP work. We first review aspects of FWPs for pedagogical purposes, then discuss connections to related works motivated by insights from neuroscience.