# Hierarchical Memory Networks

arXiv: https://arxiv.org/abs/1605.07427 · HTML: https://mlsanity.com/paper/1605.07427/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2016.

## Built on by (most cited first, 15 of 15)

- [Discrete Key-Value Bottleneck](https://mlsanity.com/paper/2207.11240/) (2022)
- [SEDM: Selective Self-Evolving Distributed Memory for Agents](https://mlsanity.com/paper/2509.09498/) (2025)
- [Sketch-GNN: Scalable Graph Neural Networks with Sublinear Training Complexity](https://mlsanity.com/paper/2406.15575/) (2024)
- [∞-former: Infinite Memory Transformer](https://mlsanity.com/paper/2109.00301/) (2021)
- [S4M: S4 for multivariate time series forecasting with Missing values](https://mlsanity.com/paper/2503.00900/) (2025)
- [A Survey of Historical Learning: Learning Models with Learning History](https://mlsanity.com/paper/2303.12992/) (2023)
- [VeTraSS: Vehicle Trajectory Similarity Search Through Graph Modeling and Representation Learning](https://mlsanity.com/paper/2404.08021/) (2024)
- [The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering](https://mlsanity.com/paper/2501.07109/) (2025)
- [Point Cloud Completion Guided by Prior Knowledge via Causal Inference](https://mlsanity.com/paper/2305.17770/) (2023)
- [Uncertainty Guided Global Memory Improves Multi-Hop Question Answering](https://mlsanity.com/paper/2311.18151/) (2023)
- [Combining Transformers with Natural Language Explanations](https://mlsanity.com/paper/2110.00125/) (2021)
- [Memory-enriched computation and learning in spiking neural networks through Hebbian plasticity](https://mlsanity.com/paper/2205.11276/) (2022)
- [SPARTAN: Sparse Hierarchical Memory for Parameter-Efficient Transformers](https://mlsanity.com/paper/2211.16634/) (2022)
- [Complexity of Symbolic Representation in Working Memory of Transformer Correlates with the Complexity of a Task](https://mlsanity.com/paper/2406.14213/) (2024)
- [ARM: Attention with Routed-Memory for Learnable Sparse Control](https://mlsanity.com/paper/2609.24417/) (2026)

## Abstract (arXiv)

Memory networks are neural networks with an explicit memory component that can be both read and written to by the network. The memory is often addressed in a soft way using a softmax function, making end-to-end training with backpropagation possible. However, this is not computationally scalable for applications which require the network to read from extremely large memories. On the other hand, it is well known that hard attention mechanisms based on reinforcement learning are challenging to train successfully. In this paper, we explore a form of hierarchical memory network, which can be considered as a hybrid between hard and soft attention memory networks. The memory is organized in a hierarchical structure such that reading from it is done with less computation than soft attention over a flat memory, while also being easier to train than hard attention over a flat memory. Specifically, we propose to incorporate Maximum Inner Product Search (MIPS) in the training and inference procedures for our hierarchical memory network. We explore the use of various state-of-the art approximate MIPS techniques and report results on SimpleQuestions, a challenging large scale factoid question answering task.