# A Taxonomy for Neural Memory Networks

arXiv: https://arxiv.org/abs/1805.00327 · HTML: https://mlsanity.com/paper/1805.00327/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2018.

## Built on by (most cited first, 5 of 5)

- [Human-inspired Perspectives: A Survey on AI Long-term Memory](https://mlsanity.com/paper/2411.00489/) (2024)
- [Memory and attention in deep learning](https://mlsanity.com/paper/2107.01390/) (2021)
- [State-Space Constraints Can Improve the Generalisation of the Differentiable Neural Computer to Input Sequences With Unseen Length](https://mlsanity.com/paper/2110.09138/) (2021)
- [Universal Recurrent Event Memories for Streaming Data](https://mlsanity.com/paper/2307.15694/) (2023)
- [External-Memory Networks for Low-Shot Learning of Targets in Forward-Looking-Sonar Imagery](https://mlsanity.com/paper/2107.10504/) (2021)

## Abstract (arXiv)

In this paper, a taxonomy for memory networks is proposed based on their memory organization. The taxonomy includes all the popular memory networks: vanilla recurrent neural network (RNN), long short term memory (LSTM ), neural stack and neural Turing machine and their variants. The taxonomy puts all these networks under a single umbrella and shows their relative expressive power , i.e. vanilla RNN <=LSTM<=neural stack<=neural RAM. The differences and commonality between these networks are analyzed. These differences are also connected to the requirements of different tasks which can give the user instructions of how to choose or design an appropriate memory network for a specific task. As a conceptual simplified class of problems, four tasks of synthetic symbol sequences: counting, counting with interference, reversing and repeat counting are developed and tested to verify our arguments. And we use two natural language processing problems to discuss how this taxonomy helps choosing the appropriate neural memory networks for real world problem.