# Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network

arXiv: https://arxiv.org/abs/2008.05441 · HTML: https://mlsanity.com/paper/2008.05441/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2020.

## Built on by (most cited first, 30 of 74)

- [FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2108.06098/) (2021)
- [Tensor Methods in Computer Vision and Deep Learning](https://mlsanity.com/paper/2107.03436/) (2021)
- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [Many-to-many Splatting for Efficient Video Frame Interpolation](https://mlsanity.com/paper/2204.03513/) (2022)
- [Layer-wise Adaptive Model Aggregation for Scalable Federated Learning](https://mlsanity.com/paper/2110.10302/) (2021)
- [Revisiting Dynamic Convolution via Matrix Decomposition](https://mlsanity.com/paper/2103.08756/) (2021)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization](https://mlsanity.com/paper/2111.14655/) (2021)
- [GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization](https://mlsanity.com/paper/2109.02220/) (2021)
- [Matrix Compression via Randomized Low Rank and Low Precision Factorization](https://mlsanity.com/paper/2310.11028/) (2023)
- [Modality-Agnostic Variational Compression of Implicit Neural Representations](https://mlsanity.com/paper/2301.09479/) (2023)
- [HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks](https://mlsanity.com/paper/2301.09422/) (2023)
- [Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction](https://mlsanity.com/paper/2202.00441/) (2022)
- [Tensor Decomposition for Model Reduction in Neural Networks: A Review](https://mlsanity.com/paper/2304.13539/) (2023)
- [Geometry-aware training of factorized layers in tensor Tucker format](https://mlsanity.com/paper/2305.19059/) (2023)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks](https://mlsanity.com/paper/2401.10341/) (2024)
- [BiPer: Binary Neural Networks using a Periodic Function](https://mlsanity.com/paper/2404.01278/) (2024)
- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [Spectral Tensor Train Parameterization of Deep Learning Layers](https://mlsanity.com/paper/2103.04217/) (2021)
- [Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization](https://mlsanity.com/paper/2112.03690/) (2021)
- [Quantization Aware Factorization for Deep Neural Network Compression](https://mlsanity.com/paper/2308.04595/) (2023)
- [Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training](https://mlsanity.com/paper/2303.13635/) (2023)
- [Harnessing Orthogonality to Train Low-Rank Neural Networks](https://mlsanity.com/paper/2401.08505/) (2024)
- [Characterizing the Accuracy -- Efficiency Trade-off of Low-rank Decomposition in Language Models](https://mlsanity.com/paper/2405.06626/) (2024)
- [Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks](https://mlsanity.com/paper/2505.20132/) (2025)
- [Semi-tensor Product-based TensorDecomposition for Neural Network Compression](https://mlsanity.com/paper/2109.15200/) (2021)
- [How to Train Unstable Looped Tensor Network](https://mlsanity.com/paper/2203.02617/) (2022)
- [Canonical convolutional neural networks](https://mlsanity.com/paper/2206.01509/) (2022)
- [SVD-NAS: Coupling Low-Rank Approximation and Neural Architecture Search](https://mlsanity.com/paper/2208.10404/) (2022)

## Abstract (arXiv)

Most state of the art deep neural networks are overparameterized and exhibit a high computational cost. A straightforward approach to this problem is to replace convolutional kernels with its low-rank tensor approximations, whereas the Canonical Polyadic tensor Decomposition is one of the most suited models. However, fitting the convolutional tensors by numerical optimization algorithms often encounters diverging components, i.e., extremely large rank-one tensors but canceling each other. Such degeneracy often causes the non-interpretable result and numerical instability for the neural network fine-tuning. This paper is the first study on degeneracy in the tensor decomposition of convolutional kernels. We present a novel method, which can stabilize the low-rank approximation of convolutional kernels and ensure efficient compression while preserving the high-quality performance of the neural networks. We evaluate our approach on popular CNN architectures for image classification and show that our method results in much lower accuracy degradation and provides consistent performance.