mlsanityData to 14 Oct 2026

On the Memory Mechanism of Tensor-Power Recurrent Models

arXiv 2103.01521 ↗First posted 2 Mar 2021cs.LG cs.SY eess.SYcited by 5

Not on the map. Why?

Abstract (arXiv)

Tensor-power (TP) recurrent model is a family of non-linear dynamical systems, of which the recurrence relation consists of a p-fold (a.k.a., degree-p) tensor product. Despite such the model frequently appears in the advanced recurrent neural networks (RNNs), to this date there is limited study on its memory property, a critical characteristic in sequence tasks. In this work, we conduct a thorough investigation of the memory mechanism of TP recurrent models. Theoretically, we prove that a large degree p is an essential condition to achieve the long memory effect, yet it would lead to unstable dynamical behaviors. Empirically, we tackle this issue by extending the degree p from discrete to a differentiable domain, such that it is efficiently learnable from a variety of datasets. Taken together, the new model is expected to benefit from the long memory effect in a stable manner. We experimentally show that the proposed model achieves competitive performance compared to various advanced RNNs in both the single-cell and seq2seq architectures.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 25 of its 49 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

25 of 49 references.

References · 25

  1. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 270
  2. 2020Learning Long-Term Dependencies in Irregularly-Sampled Time Seriescited by 83
  3. 2016Gradient Descent Learns Linear Dynamical Systemscited by 79
  4. 2018Stable Recurrent Modelscited by 66
  5. 2017Tensor-Train Recurrent Neural Networks for Video Classificationcited by 59
  6. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  7. 2020Convolutional Tensor-Train LSTM for Spatio-temporal Learningcited by 55
  8. 2020Do RNN and LSTM have Long Memory?cited by 47
  9. 2018Compressing Recurrent Neural Networks with Tensor Ring for Action Recognitioncited by 34
  10. 2017Learning Compact Recurrent Neural Networks with Block-Term Tensor Decompositioncited by 33
  11. 2013Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networkscited by 32
  12. 2017Expressive power of recurrent neural networkscited by 30
13 more
  1. 2017Compressing Recurrent Neural Network with Tensor Traincited by 28
  2. 2018Learning to Reason with Third-Order Tensor Productscited by 27
  3. 2016On Multiplicative Integration with Recurrent Neural Networkscited by 15
  4. 2018Connecting Weighted Automata and Recurrent Neural Networks through Spectral Learningcited by 15
  5. 2019Learning to Remember More with Less Memorizationcited by 14
  6. 2020Kronecker CP Decomposition with Fast Multiplication for Compressing RNNscited by 12
  7. 2016Higher Order Recurrent Neural Networkscited by 11
  8. 2019State-Regularized Recurrent Neural Networkscited by 11
  9. 2014Spectral norm of random tensorscited by 9
  10. 2019Generalized Tensor Models for Recurrent Neural Networkscited by 8
  11. 2019A Statistical Investigation of Long Memory in Language and Musiccited by 8
  12. 2020Physics-informed Tensor-train ConvLSTM for Volumetric Velocity Forecasting of Loop Currentcited by 3
  13. 2020H-OWAN: Multi-distorted Image Restoration with Tensor 1x1 Convolutioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 0; 2026: 1.

5 papers.

  1. 2022Permutation Search of Tensor Network Structures via Local Samplingcited by 13
  2. 2023Switching Autoregressive Low-rank Tensor Modelscited by 3
  3. 2024Efficient Nonparametric Tensor Decomposition for Binary and Count Datacited by 3
  4. 2023TERM Model: Tensor Ring Mixture Model for Density Estimationcited by 2
  5. 2026Recurrent Neural Networks Beyond Time: Learning from Multiple Ordered Projectionscited by 0