How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?
Not on the map. Why?
Abstract (arXiv)
Tensor decompositions have been successfully applied to compress neural networks. The compression algorithms using tensor decompositions commonly minimize the approximation error on the weights. Recent work assumes the approximation error on the weights is a proxy for the performance of the model to compress multiple layers and fine-tune the compressed model. Surprisingly, little research has systematically evaluated which approximation errors can be used to make choices regarding the layer, tensor decomposition method, and level of compression. To close this gap, we perform an experimental study to test if this assumption holds across different layers and types of decompositions, and what the effect of fine-tuning is. We include the approximation error on the features resulting from a compressed layer in our analysis to test if this provides a better proxy, as it explicitly takes the data into account. We find the approximation error on the weights has a positive correlation with the performance error, before as well as after fine-tuning. Basing the approximation error on the features does not improve the correlation significantly. While scaling the approximation error commonly is used to account for the different sizes of layers, the average correlation across layers is smaller than across all choices (i.e. layers, decompositions, and level of compression) before fine-tuning. When calculating the correlation across the different decompositions, the average rank correlation is larger than across all choices. This means multiple decompositions can be considered for compression and the approximation error can be used to choose between them.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 9 May 2023.
Builds on
21 of 33 references.
References · 21
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
- 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
- 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
- 2015Tensorizing Neural Networkscited by 250
- 2013Predicting Parameters in Deep Learningcited by 224
- 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
- 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 162
- 2016TensorLy: Tensor Learning in Pythoncited by 104
- 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 56
- 2018Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Boundscited by 43
9 more
- 2019Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimationcited by 40
- 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
- 2020Supervised Learning with Projected Entangled Pair Statescited by 31
- 2019T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensorcited by 22
- 2020Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determinationcited by 18
- 2020Hybrid Tensor Decomposition in Neural Network Compressioncited by 15
- 2021Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularizationcited by 6
- 2021Low-Rank+Sparse Tensor Compression for Neural Networkscited by 2
- 2021A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rankcited by 1
Built on by
Citing papers per year: 2025: 2.
2 papers.