Spectral Pruning for Recurrent Neural Networks
Not on the map. Why?
Abstract (arXiv)
Recurrent neural networks (RNNs) are a class of neural networks used in sequential tasks. However, in general, RNNs have a large number of parameters and involve enormous computational costs by repeating the recurrent structures in many time steps. As a method to overcome this difficulty, RNN pruning has attracted increasing attention in recent years, and it brings us benefits in terms of the reduction of computational cost as the time step progresses. However, most existing methods of RNN pruning are heuristic. The purpose of this paper is to study the theoretical scheme for RNN pruning method. We propose an appropriate pruning algorithm for RNNs inspired by "spectral pruning", and provide the generalization error bounds for compressed RNNs. We also provide numerical experiments to demonstrate our theoretical results and show the effectiveness of our pruning method compared with existing methods.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 23 May 2021.
Builds on
30 of 43 references.
References · 30
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
- 2017Spectrally-normalized margin bounds for neural networkscited by 686
- 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
- 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
- 2013Predicting Parameters in Deep Learningcited by 224
- 2014Recurrent Neural Network Regularizationcited by 189
- 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
- 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 138
- 2017SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networkscited by 135
- 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
- 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 92
- 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
18 more
- 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
- 2020Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networkscited by 40
- 2017Block-Sparse Recurrent Neural Networkscited by 39
- 2017Compressing Recurrent Neural Network with Tensor Traincited by 28
- 2021Selfish Sparse RNN Trainingcited by 26
- 2017Learning Intrinsic Sparse Structures within Long Short-Term Memorycited by 18
- 2019One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluationcited by 14
- 2015Recurrent Neural Network Training with Dark Knowledge Transfercited by 10
- 2016On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognitioncited by 10
- 2018Effective Quantization Approaches for Recurrent Neural Networkscited by 10
- 2019Compression of Recurrent Neural Networks for Efficient Language Modelingcited by 7
- 2019Structured Pruning of Recurrent Neural Networks through Neuron Selectioncited by 7
- 2019Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularizationcited by 5
- 2017Bayesian Sparsification of Recurrent Neural Networkscited by 4
- 2019RNN Architecture Learning with Sparse Regularizationcited by 4
- 2017Trace norm regularization and faster inference for embedded speech recognition RNNscited by 3
- 2019Sample Complexity Bounds for Recurrent Neural Networks with Application to Combinatorial Graph Problemscited by 2
- 2019Structured Sparsification of Gated Recurrent Neural Networkscited by 1
Built on by
Citing papers per year: 2023: 2; 2024: 1.
3 papers.