# Learning Associative Inference Using Fast Weight Memory

arXiv: https://arxiv.org/abs/2011.07831 · HTML: https://mlsanity.com/paper/2011.07831/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2020.

## Built on by (most cited first, 30 of 51)

- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention](https://mlsanity.com/paper/2404.07143/) (2024)
- [Kimi Linear: An Expressive, Efficient Attention Architecture](https://mlsanity.com/paper/2510.26692/) (2025)
- [Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues](https://mlsanity.com/paper/2411.12537/) (2024)
- [Going Beyond Linear Transformers with Recurrent Fast Weight Programmers](https://mlsanity.com/paper/2106.06295/) (2021)
- [Gated Slot Attention for Efficient Linear-Time Sequence Modeling](https://mlsanity.com/paper/2409.07146/) (2024)
- [DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products](https://mlsanity.com/paper/2502.10297/) (2025)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [End-to-End Test-Time Training for Long Context](https://mlsanity.com/paper/2512.23675/) (2025)
- [MesaNet: Sequence Modeling by Locally Optimal Test-Time Training](https://mlsanity.com/paper/2506.05233/) (2025)
- [FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention](https://mlsanity.com/paper/2108.02347/) (2021)
- [Fine-Tuning Pre-trained Transformers into Decaying Fast Weights](https://mlsanity.com/paper/2210.04243/) (2022)
- [Introducing Symmetries to Black Box Meta Reinforcement Learning](https://mlsanity.com/paper/2109.10781/) (2021)
- [Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognition](https://mlsanity.com/paper/2110.02220/) (2021)
- [Implicit Language Models are RNNs: Balancing Parallelization and Expressivity](https://mlsanity.com/paper/2502.07827/) (2025)
- [Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models](https://mlsanity.com/paper/2509.22284/) (2025)
- [Eliminating Meta Optimization Through Self-Referential Meta Learning](https://mlsanity.com/paper/2212.14392/) (2022)
- [Fast-weight Product Key Memory](https://mlsanity.com/paper/2601.00671/) (2026)
- [Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformers](https://mlsanity.com/paper/2506.00744/) (2025)
- [Adaptive Convolutions with Per-pixel Dynamic Filter Atom](https://mlsanity.com/paper/2108.07895/) (2021)
- [Consecutive Batch Model Editing with HooK Layers](https://mlsanity.com/paper/2403.05330/) (2024)
- [Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules](https://mlsanity.com/paper/2210.06184/) (2022)
- [Fast weight programming and linear transformers: from machine learning to neurobiology](https://mlsanity.com/paper/2508.08435/) (2025)
- [Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking](https://mlsanity.com/paper/2112.00086/) (2021)
- [Writing in the Margins: Better Inference Pattern for Long Context Retrieval](https://mlsanity.com/paper/2408.14906/) (2024)
- [Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions](https://mlsanity.com/paper/2410.06577/) (2024)
- [Learning State-Tracking from Code Using Linear RNNs](https://mlsanity.com/paper/2602.14814/) (2026)
- [Fixed-Point RNNs: Interpolating from Diagonal to Dense](https://mlsanity.com/paper/2503.10799/) (2025)

## Abstract (arXiv)

Humans can quickly associate stimuli to solve problems in novel contexts. Our novel neural network model learns state representations of facts that can be composed to perform such associative inference. To this end, we augment the LSTM model with an associative memory, dubbed Fast Weight Memory (FWM). Through differentiable operations at every step of a given input sequence, the LSTM updates and maintains compositional associations stored in the rapidly changing FWM weights. Our model is trained end-to-end by gradient descent and yields excellent performance on compositional language reasoning problems, meta-reinforcement-learning for POMDPs, and small-scale word-level language modelling.