# Fast Parametric Learning with Activation Memorization

arXiv: https://arxiv.org/abs/1803.10049 · HTML: https://mlsanity.com/paper/1803.10049/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2018.

## Built on by (most cited first, 9 of 9)

- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Your Transformer May Not be as Powerful as You Expect](https://mlsanity.com/paper/2205.13401/) (2022)
- [GNN-LM: Language Modeling based on Global Contexts via GNN](https://mlsanity.com/paper/2110.08743/) (2021)
- [Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture](https://mlsanity.com/paper/2310.03052/) (2023)
- [N-gram Is Back: Residual Learning of Neural Text Generation with n-gram Language Model](https://mlsanity.com/paper/2210.14431/) (2022)
- [Representation Memorization for Fast Learning New Knowledge without Forgetting](https://mlsanity.com/paper/2108.12596/) (2021)
- [Can deep learning match the efficiency of human visual long-term memory in storing object details?](https://mlsanity.com/paper/2204.13061/) (2022)
- [A Quantitative Review on Language Model Efficiency Research](https://mlsanity.com/paper/2306.01768/) (2023)
- [Memory-efficient Stochastic methods for Memory-based Transformers](https://mlsanity.com/paper/2311.08123/) (2023)

## Abstract (arXiv)

Neural networks trained with backpropagation often struggle to identify classes that have been observed a small number of times. In applications where most class labels are rare, such as language modelling, this can become a performance bottleneck. One potential remedy is to augment the network with a fast-learning non-parametric model which stores recent activations and class labels into an external memory. We explore a simplified architecture where we treat a subset of the model parameters as fast memory stores. This can help retain information over longer time intervals than a traditional memory, and does not require additional space or compute. In the case of image classification, we display faster binding of novel classes on an Omniglot image curriculum task. We also show improved performance for word-based language models on news reports (GigaWord), books (Project Gutenberg) and Wikipedia articles (WikiText-103) --- the latter achieving a state-of-the-art perplexity of 29.2.