# On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression

arXiv: https://arxiv.org/abs/1908.09982 · HTML: https://mlsanity.com/paper/1908.09982/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Aug 2019.

## Built on by (most cited first, 14 of 14)

- [FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer](https://mlsanity.com/paper/2212.03145/) (2022)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Few-Shot Bot: Prompt-Based Learning for Dialogue Systems](https://mlsanity.com/paper/2110.08118/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [Matrix Compression via Randomized Low Rank and Low Precision Factorization](https://mlsanity.com/paper/2310.11028/) (2023)
- [Shrinking Bigfoot: Reducing wav2vec 2.0 footprint](https://mlsanity.com/paper/2103.15760/) (2021)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [Exploring Extreme Parameter Compression for Pre-trained Language Models](https://mlsanity.com/paper/2205.10036/) (2022)
- [Greenformer: Factorization Toolkit for Efficient Deep Neural Networks](https://mlsanity.com/paper/2109.06762/) (2021)
- [Direction is what you need: Improving Word Embedding Compression in Large Language Models](https://mlsanity.com/paper/2106.08181/) (2021)
- [Low-Rank Compression of Language Models via Differentiable Rank Selection](https://mlsanity.com/paper/2512.13733/) (2025)
- [CarFi: Rider Localization Using Wi-Fi CSI](https://mlsanity.com/paper/2301.01592/) (2022)
- [Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models](https://mlsanity.com/paper/2302.04045/) (2023)

## Abstract (arXiv)

Despite their ubiquity in NLP tasks, Long Short-Term Memory (LSTM) networks suffer from computational inefficiencies caused by inherent unparallelizable recurrences, which further aggravates as LSTMs require more parameters for larger memory capacity. In this paper, we propose to apply low-rank matrix factorization (MF) algorithms to different recurrences in LSTMs, and explore the effectiveness on different NLP tasks and model components. We discover that additive recurrence is more important than multiplicative recurrence, and explain this by identifying meaningful correlations between matrix norms and compression performance. We compare our approach across two settings: 1) compressing core LSTM recurrences in language models, 2) compressing biLSTM layers of ELMo evaluated in three downstream NLP tasks.