# Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization

arXiv: https://arxiv.org/abs/2112.03690 · HTML: https://mlsanity.com/paper/2112.03690/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 7 Dec 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Compressing Deep Convolutional Networks using Vector Quantization](https://mlsanity.com/paper/1412.6115/) (2014)
- [Discrimination-aware Channel Pruning for Deep Neural Networks](https://mlsanity.com/paper/1810.11809/) (2018)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Compressing Neural Networks with the Hashing Trick](https://mlsanity.com/paper/1504.04788/) (2015)
- [Dynamic Channel Pruning: Feature Boosting and Suppression](https://mlsanity.com/paper/1810.05331/) (2018)
- [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015)
- [Channel Gating Neural Networks](https://mlsanity.com/paper/1805.12549/) (2018)
- [Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network](https://mlsanity.com/paper/2008.05441/) (2020)
- [More is Less: A More Complicated Network with Less Inference Complexity](https://mlsanity.com/paper/1703.08651/) (2017)
- [Towards the Limit of Network Quantization](https://mlsanity.com/paper/1612.01543/) (2016)
- [CP-decomposition with Tensor Power Method for Convolutional Neural Networks Compression](https://mlsanity.com/paper/1701.07148/) (2017)
- [Deep Neural Network Compression with Single and Multiple Level Quantization](https://mlsanity.com/paper/1803.03289/) (2018)
- [An exploration of parameter redundancy in deep networks with circulant projections](https://mlsanity.com/paper/1502.03436/) (2015)

## Built on by (most cited first, 6 of 6)

- [Position: Tensor Networks are a Valuable Asset for Green AI](https://mlsanity.com/paper/2205.12961/) (2022)
- [How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?](https://mlsanity.com/paper/2305.05318/) (2023)
- [Towards Higher Ranks via Adversarial Weight Pruning](https://mlsanity.com/paper/2311.17493/) (2023)
- [Forward and Inverse Approximation Theory for Linear Temporal Convolutional Networks](https://mlsanity.com/paper/2305.18478/) (2023)
- [Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning](https://mlsanity.com/paper/2401.08014/) (2024)
- [FGFP: A Fractional Gaussian Filter and Pruning for Deep Neural Networks Compression](https://mlsanity.com/paper/2507.22527/) (2025)

## Abstract (arXiv)

Tensor decomposition is one of the fundamental technique for model compression of deep convolution neural networks owing to its ability to reveal the latent relations among complex structures. However, most existing methods compress the networks layer by layer, which cannot provide a satisfactory solution to achieve global optimization. In this paper, we proposed a model reduction method to compress the pre-trained networks using low-rank tensor decomposition of the convolution layers. Our method is based on the optimization techniques to select the proper ranks of decomposed network layers. A new regularization method, called funnel function, is proposed to suppress the unimportant factors during the compression, so the proper ranks can be revealed much easier. The experimental results show that our algorithm can reduce more model parameters than other tensor compression methods. For ResNet18 with ImageNet2012, our reduced model can reach more than twi times speed up in terms of GMAC with merely 0.7% Top-1 accuracy drop, which outperforms most existing methods in both metrics.