# One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation

arXiv: https://arxiv.org/abs/1912.00120 · HTML: https://mlsanity.com/paper/1912.00120/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2019.

## Built on by (most cited first, 14 of 14)

- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [OPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantization](https://mlsanity.com/paper/2205.11141/) (2022)
- [Sparse Training Theory for Scalable and Efficient Agents](https://mlsanity.com/paper/2103.01636/) (2021)
- [Towards Structured Dynamic Sparse Pre-Training of BERT](https://mlsanity.com/paper/2108.06277/) (2021)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Spectral Pruning for Recurrent Neural Networks](https://mlsanity.com/paper/2105.10832/) (2021)
- [Experiments on Properties of Hidden Structures of Sparse Neural Networks](https://mlsanity.com/paper/2107.12917/) (2021)
- [One-shot Network Pruning at Initialization with Discriminative Image Patches](https://mlsanity.com/paper/2209.05683/) (2022)
- [Learning a Consensus Sub-Network with Polarization Regularization and One Pass Training](https://mlsanity.com/paper/2302.10798/) (2023)
- [RANP: Resource Aware Neuron Pruning at Initialization for 3D CNNs](https://mlsanity.com/paper/2103.08457/) (2021)
- [Graph Expansion in Pruned Recurrent Neural Network Layers Preserve Performance](https://mlsanity.com/paper/2403.11100/) (2024)
- [Rethinking Reservoir Pruning: A Dynamical Perspective for Echo State Networks](https://mlsanity.com/paper/2608.04593/) (2026)

## Abstract (arXiv)

Recent advances in the sparse neural network literature have made it possible to prune many large feed forward and convolutional networks with only a small quantity of data. Yet, these same techniques often falter when applied to the problem of recovering sparse recurrent networks. These failures are quantitative: when pruned with recent techniques, RNNs typically obtain worse performance than they do under a simple random pruning scheme. The failures are also qualitative: the distribution of active weights in a pruned LSTM or GRU network tend to be concentrated in specific neurons and gates, and not well dispersed across the entire architecture. We seek to rectify both the quantitative and qualitative issues with recurrent network pruning by introducing a new recurrent pruning objective derived from the spectrum of the recurrent Jacobian. Our objective is data efficient (requiring only 64 data points to prune the network), easy to implement, and produces 95% sparse GRUs that significantly improve on existing baselines. We evaluate on sequential MNIST, Billion Words, and Wikitext.