# Memory Capacity of Recurrent Neural Networks with Matrix Representation

arXiv: https://arxiv.org/abs/2104.07454 · HTML: https://mlsanity.com/paper/2104.07454/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 11 Apr 2021.

## Builds on (most cited first, 25 of 25)

- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)
- [A Riemannian Network for SPD Matrix Learning](https://mlsanity.com/paper/1608.04233/) (2016)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks](https://mlsanity.com/paper/1810.09536/) (2018)
- [Memory-Based Graph Networks](https://mlsanity.com/paper/2002.09518/) (2020)
- [Temporal Attention augmented Bilinear Network for Financial Time-Series Data Analysis](https://mlsanity.com/paper/1712.00975/) (2017)
- [Memory Augmented Neural Networks with Wormhole Connections](https://mlsanity.com/paper/1701.08718/) (2017)
- [Langevin-gradient parallel tempering for Bayesian neural learning](https://mlsanity.com/paper/1811.04343/) (2018)
- [The capacity of feedforward neural networks](https://mlsanity.com/paper/1901.00434/) (2019)
- [Neural Stored-program Memory](https://mlsanity.com/paper/1906.08862/) (2019)
- [Higher Order Recurrent Neural Networks](https://mlsanity.com/paper/1605.00064/) (2016)
- [Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression](https://mlsanity.com/paper/1805.10352/) (2018)
- [Graph Memory Networks for Molecular Activity Prediction](https://mlsanity.com/paper/1801.02622/) (2018)
- [Matrix Neural Networks](https://mlsanity.com/paper/1601.03805/) (2016)
- [Surrogate-assisted parallel tempering for Bayesian neural learning](https://mlsanity.com/paper/1811.08687/) (2018)
- [Constructing the Matrix Multilayer Perceptron and its Application to the VAE](https://mlsanity.com/paper/1902.01182/) (2019)

## Abstract (arXiv)

It is well known that canonical recurrent neural networks (RNNs) face limitations in learning long-term dependencies which have been addressed by memory structures in long short-term memory (LSTM) networks. Neural Turing machines (NTMs) are novel RNNs that implement the notion of programmable computers with neural network controllers that can learn simple algorithmic tasks. Matrix neural networks feature matrix representation which inherently preserves the spatial structure of data when compared to canonical neural networks that use vector-based representation. One may then argue that neural networks with matrix representations may have the potential to provide better memory capacity. In this paper, we define and study a probabilistic notion of memory capacity based on Fisher information for matrix-based RNNs. We find bounds on memory capacity for such networks under various hypotheses and compare them with their vector counterparts. In particular, we show that the memory capacity of such networks is bounded by N2 for N× N state matrix which generalizes the one known for vector networks. We also show and analyze the increase in memory capacity for such networks which is introduced when one exhibits an external state memory, such as NTMs. Consequently, we construct NTMs with RNN controllers with matrix-based representation of external memory, leading us to introduce Matrix NTMs. We demonstrate the performance of this class of memory networks under certain algorithmic learning tasks such as copying and recall and compare it with Matrix RNNs. We find an improvement in the performance of Matrix NTMs by the addition of external memory, in comparison to Matrix RNNs.