# Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes

arXiv: https://arxiv.org/abs/1610.09027 · HTML: https://mlsanity.com/paper/1610.09027/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2016.

## Built on by (most cited first, 28 of 28)

- [Recurrent Memory Transformer](https://mlsanity.com/paper/2207.06881/) (2022)
- [Scaling Transformer to 1M tokens and beyond with RMT](https://mlsanity.com/paper/2304.11062/) (2023)
- [Divide-and-Assemble: Learning Block-wise Memory for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2107.13118/) (2021)
- [Learning Fast and Slow for Online Time Series Forecasting](https://mlsanity.com/paper/2202.11672/) (2022)
- [Cross-Domain Video Anomaly Detection without Target Domain Adaptation](https://mlsanity.com/paper/2212.07010/) (2022)
- [Fast-weight Product Key Memory](https://mlsanity.com/paper/2601.00671/) (2026)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity](https://mlsanity.com/paper/2607.07386/) (2026)
- [Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture](https://mlsanity.com/paper/2310.03052/) (2023)
- [MeSH: Memory-as-State-Highways for Recursive Transformers](https://mlsanity.com/paper/2510.07739/) (2025)
- [∞-former: Infinite Memory Transformer](https://mlsanity.com/paper/2109.00301/) (2021)
- [Learning to Rehearse in Long Sequence Memorization](https://mlsanity.com/paper/2106.01096/) (2021)
- [A Theoretical View on Sparsely Activated Networks](https://mlsanity.com/paper/2208.04461/) (2022)
- [Memory-Assisted Sub-Prototype Mining for Universal Domain Adaptation](https://mlsanity.com/paper/2310.05453/) (2023)
- [Neural Sequence-to-grid Module for Learning Symbolic Rules](https://mlsanity.com/paper/2101.04921/) (2021)
- [An Evolved Universal Transformer Memory](https://mlsanity.com/paper/2410.13166/) (2024)
- [A Brain-inspired Memory Transformation based Differentiable Neural Computer for Reasoning-based Question Answering](https://mlsanity.com/paper/2301.02809/) (2023)
- [A Framework for Inference Inspired by Human Memory Mechanisms](https://mlsanity.com/paper/2310.09297/) (2023)
- [Token Turing Machines are Efficient Vision Models](https://mlsanity.com/paper/2409.07613/) (2024)
- [State-Space Constraints Can Improve the Generalisation of the Differentiable Neural Computer to Input Sequences With Unseen Length](https://mlsanity.com/paper/2110.09138/) (2021)
- [Clear Memory-Augmented Auto-Encoder for Surface Defect Detection](https://mlsanity.com/paper/2208.03879/) (2022)
- [Anomaly Detection in Aeronautics Data with Quantum-compatible Discrete Deep Generative Model](https://mlsanity.com/paper/2303.12302/) (2023)
- [Complexity of Symbolic Representation in Working Memory of Transformer Correlates with the Complexity of a Task](https://mlsanity.com/paper/2406.14213/) (2024)
- [Online Vector Quantized Attention](https://mlsanity.com/paper/2602.03922/) (2026)
- [Differentiable Neural Computers with Memory Demon](https://mlsanity.com/paper/2211.02987/) (2022)
- [DNCs Require More Planning Steps](https://mlsanity.com/paper/2406.02187/) (2024)
- [Discrete Dictionary-based Decomposition Layer for Structured Representation Learning](https://mlsanity.com/paper/2406.06976/) (2024)
- [Epistemic Memory: A Validity Layer for Self-Maintaining Intelligent Systems](https://mlsanity.com/paper/2510.16899/) (2025)

## Abstract (arXiv)

Neural networks augmented with external memory have the ability to learn algorithmic solutions to complex tasks. These models appear promising for applications such as language modeling and machine translation. However, they scale poorly in both space and time as the amount of memory grows --- limiting their applicability to real-world domains. Here, we present an end-to-end differentiable memory access scheme, which we call Sparse Access Memory (SAM), that retains the representational power of the original approaches whilst training efficiently with very large memories. We show that SAM achieves asymptotic lower bounds in space and time complexity, and find that an implementation runs 1,!000× faster and with 3,!000× less physical memory than non-sparse models. SAM learns with comparable data efficiency to existing models on a range of synthetic tasks and one-shot Omniglot character recognition, and can scale to tasks requiring 100,!000s of time steps and memories. As well, we show how our approach can be adapted for models that maintain temporal associations between memories, as with the recently introduced Differentiable Neural Computer.