# Tensor Decomposition for Compressing Recurrent Neural Network

arXiv: https://arxiv.org/abs/1802.10410 · HTML: https://mlsanity.com/paper/1802.10410/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2018.

## Built on by (most cited first, 10 of 10)

- [Annealing Knowledge Distillation](https://mlsanity.com/paper/2104.07163/) (2021)
- [HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compression](https://mlsanity.com/paper/2211.16749/) (2022)
- [Towards Extremely Compact RNNs for Video Recognition with Fully Decomposed Hierarchical Tucker Structure](https://mlsanity.com/paper/2104.05758/) (2021)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [Semi-tensor Product-based TensorDecomposition for Neural Network Compression](https://mlsanity.com/paper/2109.15200/) (2021)
- [Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization](https://mlsanity.com/paper/2212.05998/) (2022)
- [A Tensor Decomposition Perspective on Second-order RNNs](https://mlsanity.com/paper/2406.05045/) (2024)
- [Compressive Sensing with Tensorized Autoencoder](https://mlsanity.com/paper/2303.06235/) (2023)
- [Improved knowledge distillation by utilizing backward pass knowledge in neural networks](https://mlsanity.com/paper/2301.12006/) (2023)
- [Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization](https://mlsanity.com/paper/2507.01841/) (2025)

## Abstract (arXiv)

In the machine learning fields, Recurrent Neural Network (RNN) has become a popular architecture for sequential data modeling. However, behind the impressive performance, RNNs require a large number of parameters for both training and inference. In this paper, we are trying to reduce the number of parameters and maintain the expressive power from RNN simultaneously. We utilize several tensor decompositions method including CANDECOMP/PARAFAC (CP), Tucker decomposition and Tensor Train (TT) to re-parameterize the Gated Recurrent Unit (GRU) RNN. We evaluate all tensor-based RNNs performance on sequence modeling tasks with a various number of parameters. Based on our experiment results, TT-GRU achieved the best results in a various number of parameters compared to other decomposition methods.