# Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs

arXiv: https://arxiv.org/abs/2008.09342 · HTML: https://mlsanity.com/paper/2008.09342/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2020.

## Built on by (most cited first, 12 of 12)

- [Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips](https://mlsanity.com/paper/2404.03663/) (2024)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions](https://mlsanity.com/paper/2110.04176/) (2021)
- [DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Models](https://mlsanity.com/paper/2402.17412/) (2024)
- [tntorch: Tensor Network Learning with PyTorch](https://mlsanity.com/paper/2206.11128/) (2022)
- [On the Memory Mechanism of Tensor-Power Recurrent Models](https://mlsanity.com/paper/2103.01521/) (2021)
- [Probabilistic Modeling: Proving the Lottery Ticket Hypothesis in Spiking Neural Network](https://mlsanity.com/paper/2305.12148/) (2023)
- [A Tensor Decomposition Perspective on Second-order RNNs](https://mlsanity.com/paper/2406.05045/) (2024)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [Lite it fly: An All-Deformable-Butterfly Network](https://mlsanity.com/paper/2311.08125/) (2023)
- [Convolutional Neural Network Compression Based on Low-Rank Decomposition](https://mlsanity.com/paper/2408.16289/) (2024)
- [Lightweight Temporal Transformer Decomposition for Federated Autonomous Driving](https://mlsanity.com/paper/2506.23523/) (2025)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are powerful in the tasks oriented to sequential data, such as natural language processing and video recognition. However, since the modern RNNs, including long-short term memory (LSTM) and gated recurrent unit (GRU) networks, have complex topologies and expensive space/computation complexity, compressing them becomes a hot and promising topic in recent years. Among plenty of compression methods, tensor decomposition, e.g., tensor train (TT), block term (BT), tensor ring (TR) and hierarchical Tucker (HT), appears to be the most amazing approach since a very high compression ratio might be obtained. Nevertheless, none of these tensor decomposition formats can provide both the space and computation efficiency. In this paper, we consider to compress RNNs based on a novel Kronecker CANDECOMP/PARAFAC (KCP) decomposition, which is derived from Kronecker tensor (KT) decomposition, by proposing two fast algorithms of multiplication between the input and the tensor-decomposed weight. According to our experiments based on UCF11, Youtube Celebrities Face and UCF50 datasets, it can be verified that the proposed KCP-RNNs have comparable performance of accuracy with those in other tensor-decomposed formats, and even 278,219x compression ratio could be obtained by the low rank KCP. More importantly, KCP-RNNs are efficient in both space and computation complexity compared with other tensor-decomposed ones under similar ranks. Besides, we find KCP has the best potential for parallel computing to accelerate the calculations in neural networks.