# How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?

arXiv: https://arxiv.org/abs/2305.05318 · HTML: https://mlsanity.com/paper/2305.05318/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [TensorLy: Tensor Learning in Python](https://mlsanity.com/paper/1610.09555/) (2016)
- [Ultimate tensorization: compressing convolutional and FC layers alike](https://mlsanity.com/paper/1611.03214/) (2016)
- [Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds](https://mlsanity.com/paper/1804.05345/) (2018)
- [Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation](https://mlsanity.com/paper/1906.06196/) (2019)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [Supervised Learning with Projected Entangled Pair States](https://mlsanity.com/paper/2009.09932/) (2020)
- [T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor](https://mlsanity.com/paper/1904.02698/) (2019)
- [Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination](https://mlsanity.com/paper/2010.08689/) (2020)
- [Hybrid Tensor Decomposition in Neural Network Compression](https://mlsanity.com/paper/2006.15938/) (2020)
- [Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization](https://mlsanity.com/paper/2112.03690/) (2021)
- [Low-Rank+Sparse Tensor Compression for Neural Networks](https://mlsanity.com/paper/2111.01697/) (2021)
- [A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rank](https://mlsanity.com/paper/2111.15179/) (2021)

## Built on by (most cited first, 2 of 2)

- [Efficient CNN Compression via Multi-method Low Rank Factorization and Feature Map Similarity](https://mlsanity.com/paper/2510.00062/) (2025)
- [Tensor-Efficient High-Dimensional Q-learning](https://mlsanity.com/paper/2511.03595/) (2025)

## Abstract (arXiv)

Tensor decompositions have been successfully applied to compress neural networks. The compression algorithms using tensor decompositions commonly minimize the approximation error on the weights. Recent work assumes the approximation error on the weights is a proxy for the performance of the model to compress multiple layers and fine-tune the compressed model. Surprisingly, little research has systematically evaluated which approximation errors can be used to make choices regarding the layer, tensor decomposition method, and level of compression. To close this gap, we perform an experimental study to test if this assumption holds across different layers and types of decompositions, and what the effect of fine-tuning is. We include the approximation error on the features resulting from a compressed layer in our analysis to test if this provides a better proxy, as it explicitly takes the data into account. We find the approximation error on the weights has a positive correlation with the performance error, before as well as after fine-tuning. Basing the approximation error on the features does not improve the correlation significantly. While scaling the approximation error commonly is used to account for the different sizes of layers, the average correlation across layers is smaller than across all choices (i.e. layers, decompositions, and level of compression) before fine-tuning. When calculating the correlation across the different decompositions, the average rank correlation is larger than across all choices. This means multiple decompositions can be considered for compression and the approximation error can be used to choose between them.