# Learning Compact Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1604.02594 · HTML: https://mlsanity.com/paper/1604.02594/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2016.

## Built on by (most cited first, 6 of 6)

- [D-TensoRF: Tensorial Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2212.02375/) (2022)
- [End-to-End Neural Network Compression via fracℓ1ℓ2 Regularized Latency Surrogates](https://mlsanity.com/paper/2306.05785/) (2023)
- [Compression of Recurrent Neural Networks using Matrix Factorization](https://mlsanity.com/paper/2310.12688/) (2023)
- [IMPACT: Importance-Aware Activation Space Reconstruction](https://mlsanity.com/paper/2507.03828/) (2025)
- [Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training](https://mlsanity.com/paper/2106.12089/) (2021)
- [Linear Reusable Neural Bases Architecture for Network Compression](https://mlsanity.com/paper/2609.01550/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs), including long short-term memory (LSTM) RNNs, have produced state-of-the-art results on a variety of speech recognition tasks. However, these models are often too large in size for deployment on mobile devices with memory and latency constraints. In this work, we study mechanisms for learning compact RNNs and LSTMs via low-rank factorizations and parameter sharing schemes. Our goal is to investigate redundancies in recurrent architectures where compression can be admitted without losing performance. A hybrid strategy of using structured matrices in the bottom layers and shared low-rank factors on the top layers is found to be particularly effective, reducing the parameters of a standard LSTM by 75%, at a small cost of 0.3% increase in WER, on a 2,000-hr English Voice Search task.