# Compressing Recurrent Neural Network with Tensor Train

arXiv: https://arxiv.org/abs/1705.08052 · HTML: https://mlsanity.com/paper/1705.08052/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2017.

## Built on by (most cited first, 28 of 28)

- [Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations](https://mlsanity.com/paper/2205.13571/) (2022)
- [Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators](https://mlsanity.com/paper/2106.02205/) (2021)
- [CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation](https://mlsanity.com/paper/2502.10940/) (2025)
- [QTN-VQC: An End-to-End Learning framework for Quantum Neural Networks](https://mlsanity.com/paper/2110.03861/) (2021)
- [Tensor networks for interpretable and efficient quantum-inspired machine learning](https://mlsanity.com/paper/2311.11258/) (2023)
- [CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization](https://mlsanity.com/paper/2405.14377/) (2024)
- [Large-Scale Learning with Fourier Features and Tensor Decompositions](https://mlsanity.com/paper/2109.01545/) (2021)
- [Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling](https://mlsanity.com/paper/2205.14283/) (2022)
- [Guaranteed Nonconvex Factorization Approach for Tensor Train Recovery](https://mlsanity.com/paper/2401.02592/) (2024)
- [Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees](https://mlsanity.com/paper/2406.17887/) (2024)
- [Tensor-Train Recurrent Neural Networks for Interpretable Multi-Way Financial Forecasting](https://mlsanity.com/paper/2105.04983/) (2021)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [On the Memory Mechanism of Tensor-Power Recurrent Models](https://mlsanity.com/paper/2103.01521/) (2021)
- [A Scalable Factorization Approach for High-Order Structured Tensor Recovery](https://mlsanity.com/paper/2506.16032/) (2025)
- [3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration](https://mlsanity.com/paper/2105.06250/) (2021)
- [Spectral Pruning for Recurrent Neural Networks](https://mlsanity.com/paper/2105.10832/) (2021)
- [Position: Tensor Networks are a Valuable Asset for Green AI](https://mlsanity.com/paper/2205.12961/) (2022)
- [Canonical convolutional neural networks](https://mlsanity.com/paper/2206.01509/) (2022)
- [Tetra-AML: Automatic Machine Learning via Tensor Networks](https://mlsanity.com/paper/2303.16214/) (2023)
- [Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT](https://mlsanity.com/paper/2603.28534/) (2026)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [TTML: tensor trains for general supervised machine learning](https://mlsanity.com/paper/2203.04352/) (2022)
- [Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding](https://mlsanity.com/paper/2306.01076/) (2023)
- [Error Analysis of Tensor-Train Cross Approximation](https://mlsanity.com/paper/2207.04327/) (2022)
- [Rec-AD: An Efficient Computation Framework for FDIA Detection Based on Tensor Train Decomposition and Deep Learning Recommendation Model](https://mlsanity.com/paper/2507.14668/) (2025)
- [Embedding Compression for Text Classification Using Dictionary Screening](https://mlsanity.com/paper/2211.12715/) (2022)
- [Quantum-inspired tensor networks in machine learning models](https://mlsanity.com/paper/2604.14287/) (2026)
- [Entropy-Guided Tensor Compression for Multimodal Federated Learning on Edge Devices](https://mlsanity.com/paper/2607.06651/) (2026)

## Abstract (arXiv)

Recurrent Neural Network (RNN) are a popular choice for modeling temporal and sequential tasks and achieve many state-of-the-art performance on various complex problems. However, most of the state-of-the-art RNNs have millions of parameters and require many computational resources for training and predicting new data. This paper proposes an alternative RNN model to reduce the number of parameters significantly by representing the weight parameters based on Tensor Train (TT) format. In this paper, we implement the TT-format representation for several RNN architectures such as simple RNN and Gated Recurrent Unit (GRU). We compare and evaluate our proposed RNN model with uncompressed RNN model on sequence classification and sequence prediction tasks. Our proposed RNNs with TT-format are able to preserve the performance while reducing the number of RNN parameters significantly up to 40 times smaller.