# Compression of Recurrent Neural Networks using Matrix Factorization

arXiv: https://arxiv.org/abs/2310.12688 · HTML: https://mlsanity.com/paper/2310.12688/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 19 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Compression-aware Training of Deep Networks](https://mlsanity.com/paper/1711.02638/) (2017)
- [Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks](https://mlsanity.com/paper/1312.4461/) (2013)
- [TRP: Trained Rank Pruning for Efficient Deep Neural Networks](https://mlsanity.com/paper/2004.14566/) (2020)
- [A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions](https://mlsanity.com/paper/2010.03954/) (2020)
- [Compression of Recurrent Neural Networks for Efficient Language Modeling](https://mlsanity.com/paper/1902.02380/) (2019)
- [Learning Compact Recurrent Neural Networks](https://mlsanity.com/paper/1604.02594/) (2016)
- [Decomposable-Net: Scalable Low-Rank Compression for Neural Networks](https://mlsanity.com/paper/1910.13141/) (2019)
- [A light neural network for modulation detection under impairments](https://mlsanity.com/paper/2003.12260/) (2020)

## Built on by (most cited first, 1 of 1)

- [On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization](https://mlsanity.com/paper/2507.09428/) (2025)

## Abstract (arXiv)

Compressing neural networks is a key step when deploying models for real-time or embedded applications. Factorizing the model's matrices using low-rank approximations is a promising method for achieving compression. While it is possible to set the rank before training, this approach is neither flexible nor optimal. In this work, we propose a post-training rank-selection method called Rank-Tuning that selects a different rank for each matrix. Used in combination with training adaptations, our method achieves high compression rates with no or little performance degradation. Our numerical experiments on signal processing tasks show that we can compress recurrent neural networks up to 14x with at most 1.4% relative performance reduction.