# On the Memory Mechanism of Tensor-Power Recurrent Models

arXiv: https://arxiv.org/abs/2103.01521 · HTML: https://mlsanity.com/paper/2103.01521/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 25 of 25)

- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [Learning Long-Term Dependencies in Irregularly-Sampled Time Series](https://mlsanity.com/paper/2006.04418/) (2020)
- [Gradient Descent Learns Linear Dynamical Systems](https://mlsanity.com/paper/1609.05191/) (2016)
- [Stable Recurrent Models](https://mlsanity.com/paper/1805.10369/) (2018)
- [Tensor-Train Recurrent Neural Networks for Video Classification](https://mlsanity.com/paper/1707.01786/) (2017)
- [Learning Longer-term Dependencies in RNNs with Auxiliary Losses](https://mlsanity.com/paper/1803.00144/) (2018)
- [Convolutional Tensor-Train LSTM for Spatio-temporal Learning](https://mlsanity.com/paper/2002.09131/) (2020)
- [Do RNN and LSTM have Long Memory?](https://mlsanity.com/paper/2006.03860/) (2020)
- [Compressing Recurrent Neural Networks with Tensor Ring for Action Recognition](https://mlsanity.com/paper/1811.07503/) (2018)
- [Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition](https://mlsanity.com/paper/1712.05134/) (2017)
- [Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks](https://mlsanity.com/paper/1311.1780/) (2013)
- [Expressive power of recurrent neural networks](https://mlsanity.com/paper/1711.00811/) (2017)
- [Compressing Recurrent Neural Network with Tensor Train](https://mlsanity.com/paper/1705.08052/) (2017)
- [Learning to Reason with Third-Order Tensor Products](https://mlsanity.com/paper/1811.12143/) (2018)
- [On Multiplicative Integration with Recurrent Neural Networks](https://mlsanity.com/paper/1606.06630/) (2016)
- [Connecting Weighted Automata and Recurrent Neural Networks through Spectral Learning](https://mlsanity.com/paper/1807.01406/) (2018)
- [Learning to Remember More with Less Memorization](https://mlsanity.com/paper/1901.01347/) (2019)
- [Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs](https://mlsanity.com/paper/2008.09342/) (2020)
- [Higher Order Recurrent Neural Networks](https://mlsanity.com/paper/1605.00064/) (2016)
- [State-Regularized Recurrent Neural Networks](https://mlsanity.com/paper/1901.08817/) (2019)
- [Spectral norm of random tensors](https://mlsanity.com/paper/1407.1870/) (2014)
- [Generalized Tensor Models for Recurrent Neural Networks](https://mlsanity.com/paper/1901.10801/) (2019)
- [A Statistical Investigation of Long Memory in Language and Music](https://mlsanity.com/paper/1904.03834/) (2019)
- [Physics-informed Tensor-train ConvLSTM for Volumetric Velocity Forecasting of Loop Current](https://mlsanity.com/paper/2008.01798/) (2020)
- [H-OWAN: Multi-distorted Image Restoration with Tensor 1x1 Convolution](https://mlsanity.com/paper/2001.10853/) (2020)

## Built on by (most cited first, 5 of 5)

- [Permutation Search of Tensor Network Structures via Local Sampling](https://mlsanity.com/paper/2206.06597/) (2022)
- [Switching Autoregressive Low-rank Tensor Models](https://mlsanity.com/paper/2306.03291/) (2023)
- [Efficient Nonparametric Tensor Decomposition for Binary and Count Data](https://mlsanity.com/paper/2401.07711/) (2024)
- [TERM Model: Tensor Ring Mixture Model for Density Estimation](https://mlsanity.com/paper/2312.08075/) (2023)
- [Recurrent Neural Networks Beyond Time: Learning from Multiple Ordered Projections](https://mlsanity.com/paper/2608.09690/) (2026)

## Abstract (arXiv)

Tensor-power (TP) recurrent model is a family of non-linear dynamical systems, of which the recurrence relation consists of a p-fold (a.k.a., degree-p) tensor product. Despite such the model frequently appears in the advanced recurrent neural networks (RNNs), to this date there is limited study on its memory property, a critical characteristic in sequence tasks. In this work, we conduct a thorough investigation of the memory mechanism of TP recurrent models. Theoretically, we prove that a large degree p is an essential condition to achieve the long memory effect, yet it would lead to unstable dynamical behaviors. Empirically, we tackle this issue by extending the degree p from discrete to a differentiable domain, such that it is efficiently learnable from a variety of datasets. Taken together, the new model is expected to benefit from the long memory effect in a stable manner. We experimentally show that the proposed model achieves competitive performance compared to various advanced RNNs in both the single-cell and seq2seq architectures.