# Spectral Pruning for Recurrent Neural Networks

arXiv: https://arxiv.org/abs/2105.10832 · HTML: https://mlsanity.com/paper/2105.10832/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 23 May 2021.

## Builds on (most cited first, 30 of 30)

- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Recurrent Neural Network Regularization](https://mlsanity.com/paper/1409.2329/) (2014)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://mlsanity.com/paper/1512.05287/) (2015)
- [On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions](https://mlsanity.com/paper/1502.06800/) (2015)
- [SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks](https://mlsanity.com/paper/1708.04485/) (2017)
- [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://mlsanity.com/paper/1504.00941/) (2015)
- [Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling](https://mlsanity.com/paper/1611.01462/) (2016)
- [Exploring Sparsity in Recurrent Neural Networks](https://mlsanity.com/paper/1704.05119/) (2017)
- [On Generalization Bounds of a Family of Recurrent Neural Networks](https://mlsanity.com/paper/1910.12947/) (2019)
- [Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks](https://mlsanity.com/paper/2012.08749/) (2020)
- [Block-Sparse Recurrent Neural Networks](https://mlsanity.com/paper/1711.02782/) (2017)
- [Compressing Recurrent Neural Network with Tensor Train](https://mlsanity.com/paper/1705.08052/) (2017)
- [Selfish Sparse RNN Training](https://mlsanity.com/paper/2101.09048/) (2021)
- [Learning Intrinsic Sparse Structures within Long Short-Term Memory](https://mlsanity.com/paper/1709.05027/) (2017)
- [One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation](https://mlsanity.com/paper/1912.00120/) (2019)
- [Recurrent Neural Network Training with Dark Knowledge Transfer](https://mlsanity.com/paper/1505.04630/) (2015)
- [On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition](https://mlsanity.com/paper/1603.08042/) (2016)
- [Effective Quantization Approaches for Recurrent Neural Networks](https://mlsanity.com/paper/1802.02615/) (2018)
- [Compression of Recurrent Neural Networks for Efficient Language Modeling](https://mlsanity.com/paper/1902.02380/) (2019)
- [Structured Pruning of Recurrent Neural Networks through Neuron Selection](https://mlsanity.com/paper/1906.06847/) (2019)
- [Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization](https://mlsanity.com/paper/1904.04163/) (2019)
- [Bayesian Sparsification of Recurrent Neural Networks](https://mlsanity.com/paper/1708.00077/) (2017)
- [RNN Architecture Learning with Sparse Regularization](https://mlsanity.com/paper/1909.03011/) (2019)
- [Trace norm regularization and faster inference for embedded speech recognition RNNs](https://mlsanity.com/paper/1710.09026/) (2017)
- [Sample Complexity Bounds for Recurrent Neural Networks with Application to Combinatorial Graph Problems](https://mlsanity.com/paper/1901.10289/) (2019)
- [Structured Sparsification of Gated Recurrent Neural Networks](https://mlsanity.com/paper/1911.05585/) (2019)

## Built on by (most cited first, 3 of 3)

- [On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats](https://mlsanity.com/paper/2301.01651/) (2023)
- [Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers](https://mlsanity.com/paper/2305.18382/) (2023)
- [Graph Expansion in Pruned Recurrent Neural Network Layers Preserve Performance](https://mlsanity.com/paper/2403.11100/) (2024)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are a class of neural networks used in sequential tasks. However, in general, RNNs have a large number of parameters and involve enormous computational costs by repeating the recurrent structures in many time steps. As a method to overcome this difficulty, RNN pruning has attracted increasing attention in recent years, and it brings us benefits in terms of the reduction of computational cost as the time step progresses. However, most existing methods of RNN pruning are heuristic. The purpose of this paper is to study the theoretical scheme for RNN pruning method. We propose an appropriate pruning algorithm for RNNs inspired by "spectral pruning", and provide the generalization error bounds for compressed RNNs. We also provide numerical experiments to demonstrate our theoretical results and show the effectiveness of our pruning method compared with existing methods.