# Compression of Recurrent Neural Networks for Efficient Language Modeling

arXiv: https://arxiv.org/abs/1902.02380 · HTML: https://mlsanity.com/paper/1902.02380/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2019.

## Built on by (most cited first, 7 of 7)

- [Integration of Large Language Models and Federated Learning](https://mlsanity.com/paper/2307.08925/) (2023)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [Spectral Pruning for Recurrent Neural Networks](https://mlsanity.com/paper/2105.10832/) (2021)
- [Sparse Deep Learning for Time Series Data: Theory and Applications](https://mlsanity.com/paper/2310.03243/) (2023)
- [A Review of Hybrid and Ensemble in Deep Learning for Natural Language Processing](https://mlsanity.com/paper/2312.05589/) (2023)
- [Compression of Recurrent Neural Networks using Matrix Factorization](https://mlsanity.com/paper/2310.12688/) (2023)
- [Embedding Compression for Text Classification Using Dictionary Screening](https://mlsanity.com/paper/2211.12715/) (2022)

## Abstract (arXiv)

Recurrent neural networks have proved to be an effective method for statistical language modeling. However, in practice their memory and run-time complexity are usually too large to be implemented in real-time offline mobile applications. In this paper we consider several compression techniques for recurrent neural networks including Long-Short Term Memory models. We make particular attention to the high-dimensional output problem caused by the very large vocabulary size. We focus on effective compression methods in the context of their exploitation on devices: pruning, quantization, and matrix decomposition approaches (low-rank factorization and tensor train decomposition, in particular). For each model we investigate the trade-off between its size, suitability for fast inference and perplexity. We propose a general pipeline for applying the most suitable methods to compress recurrent neural networks for language modeling. It has been shown in the experimental study with the Penn Treebank (PTB) dataset that the most efficient results in terms of speed and compression-perplexity balance are obtained by matrix decomposition techniques.