# On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition

arXiv: https://arxiv.org/abs/1603.08042 · HTML: https://mlsanity.com/paper/1603.08042/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2016.

## Built on by (most cited first, 10 of 10)

- [Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models](https://mlsanity.com/paper/2110.03334/) (2021)
- [CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition](https://mlsanity.com/paper/2106.07734/) (2021)
- [Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency](https://mlsanity.com/paper/2104.02176/) (2021)
- [Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods](https://mlsanity.com/paper/2402.04051/) (2024)
- [Spectral Pruning for Recurrent Neural Networks](https://mlsanity.com/paper/2105.10832/) (2021)
- [Sub 8-Bit Quantization of Streaming Keyword Spotting Models for Embedded Chipsets](https://mlsanity.com/paper/2207.06920/) (2022)
- [Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization](https://mlsanity.com/paper/2105.01868/) (2021)
- [SEOFP-NET: Compression and Acceleration of Deep Neural Networks for Speech Enhancement Using Sign-Exponent-Only Floating-Points](https://mlsanity.com/paper/2111.04436/) (2021)
- [Deep learning model compression using network sensitivity and gradients](https://mlsanity.com/paper/2210.05111/) (2022)
- [Embedding Compression for Text Classification Using Dictionary Screening](https://mlsanity.com/paper/2211.12715/) (2022)

## Abstract (arXiv)

We study the problem of compressing recurrent neural networks (RNNs). In particular, we focus on the compression of RNN acoustic models, which are motivated by the goal of building compact and accurate speech recognition systems which can be run efficiently on mobile devices. In this work, we present a technique for general recurrent model compression that jointly compresses both recurrent and non-recurrent inter-layer weight matrices. We find that the proposed technique allows us to reduce the size of our Long Short-Term Memory (LSTM) acoustic model to a third of its original size with negligible loss in accuracy.