# Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling

arXiv: https://arxiv.org/abs/2009.01803 · HTML: https://mlsanity.com/paper/2009.01803/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2020.

## Built on by (most cited first, 5 of 5)

- [Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognition](https://mlsanity.com/paper/2110.02220/) (2021)
- [Fast weight programming and linear transformers: from machine learning to neurobiology](https://mlsanity.com/paper/2508.08435/) (2025)
- [Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models](https://mlsanity.com/paper/2403.19709/) (2024)
- [Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networks](https://mlsanity.com/paper/2211.09440/) (2022)
- [TACOS: Task Agnostic Continual Learning in Spiking Neural Networks](https://mlsanity.com/paper/2409.00021/) (2024)

## Abstract (arXiv)

Training a deep neural network requires a large amount of single-task data and involves a long time-consuming optimization phase. This is not scalable to complex, realistic environments with new unexpected changes. Humans can perform fast incremental learning on the fly and memory systems in the brain play a critical role. We introduce Sparse Meta Networks -- a meta-learning approach to learn online sequential adaptation algorithms for deep neural networks, by using deep neural networks. We augment a deep neural network with a layer-specific fast-weight memory. The fast-weights are generated sparsely at each time step and accumulated incrementally through time providing a useful inductive bias for online continual adaptation. We demonstrate strong performance on a variety of sequential adaptation scenarios, from a simple online reinforcement learning to a large scale adaptive language modelling.