mlsanityData to 7 Oct 2026

A Tensor Decomposition Perspective on Second-order RNNs

arXiv 2406.05045 ↗First posted 7 Jun 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Second-order Recurrent Neural Networks (2RNNs) extend RNNs by leveraging second-order interactions for sequence modelling. These models are provably more expressive than their first-order counterparts and have connections to well-studied models from formal language theory. However, their large parameter tensor makes computations intractable. To circumvent this issue, one approach known as MIRNN consists in limiting the type of interactions used by the model. Another is to leverage tensor decomposition to diminish the parameter count. In this work, we study the model resulting from parameterizing 2RNNs using the CP decomposition, which we call CPRNN. Intuitively, the rank of the decomposition should reduce expressivity. We analyze how rank and hidden size affect model capacity and show the relationships between RNNs, 2RNNs, MIRNNs, and CPRNNs based on these parameters. We support these results empirically with experiments on the Penn Treebank dataset which demonstrate that, with a fixed parameter budget, CPRNNs outperforms RNNs, 2RNNs, and MIRNNs with the right choice of rank and hidden size.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 33 of its 63 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

33 of 63 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  4. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  5. 2022Neural Networks and the Chomsky Hierarchycited by 214
  6. 2016Tensor Ring Decompositioncited by 125
  7. 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
  8. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  9. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 65
  10. 2020A Formal Hierarchy of RNN Architecturescited by 61
  11. 2017Tensor-Train Recurrent Neural Networks for Video Classificationcited by 59
  12. 2019Sequential Neural Networks as Automatacited by 54
21 more
  1. 2016Inductive Bias of Deep Convolutional Networks through Pooling Geometrycited by 53
  2. 2019On the Computational Power of RNNscited by 35
  3. 2017Learning Compact Recurrent Neural Networks with Block-Term Tensor Decompositioncited by 33
  4. 2016Convolutional Rectifier Networks as Generalized Tensor Decompositionscited by 32
  5. 2017Expressive power of recurrent neural networkscited by 30
  6. 2017Recurrent Neural Networks as Weighted Language Recognizerscited by 30
  7. 2016Multiplicative LSTM for sequence modellingcited by 27
  8. 2022Scalable Interpretability via Polynomialscited by 25
  9. 2016On Multiplicative Integration with Recurrent Neural Networkscited by 15
  10. 2017On the Long-Term Memory of Deep Recurrent Networkscited by 14
  11. 2020Kronecker CP Decomposition with Fast Multiplication for Compressing RNNscited by 12
  12. 2022Finite Sample Identification of Bilinear Dynamical Systemscited by 12
  13. 2022On the Ability of Graph Neural Networks to Model Interactions Between Verticescited by 12
  14. 2024Multilinear Operator Networkscited by 12
  15. 2018Tensor Decomposition for Compressing Recurrent Neural Networkcited by 10
  16. 2019Generalized Tensor Models for Recurrent Neural Networkscited by 8
  17. 2020Connecting Weighted Automata, Tensor Networks and Recurrent Neural Networks through Spectral Learningcited by 7
  18. 2017Gated Recurrent Neural Tensor Networkcited by 4
  19. 2023What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglementcited by 4
  20. 2024Language Modeling Using Tensor Trainscited by 3
  21. 2014Modeling Compositionality with Multiplicative Recurrent Neural Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  2. 2025Low-Rank Tensor Decompositions for the Theory of Neural Networkscited by 1
  3. 2026On the Role of Depth in the Expressivity of RNNscited by 0