# Convolutional Neural Network Compression Based on Low-Rank Decomposition

arXiv: https://arxiv.org/abs/2408.16289 · HTML: https://mlsanity.com/paper/2408.16289/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 93 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 20 of 20)

- [Class-incremental learning: survey and performance evaluation on image classification](https://mlsanity.com/paper/2010.15277/) (2020)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Towards Large-Scale Small Object Detection: Survey and Benchmarks](https://mlsanity.com/paper/2207.14096/) (2022)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.10671/) (2020)
- [FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization](https://mlsanity.com/paper/2403.06908/) (2024)
- [Blindly Assess Quality of In-the-Wild Videos via Quality-aware Pre-training and Motion Perception](https://mlsanity.com/paper/2108.08505/) (2021)
- [SCOP: Scientific Control for Reliable Neural Network Pruning](https://mlsanity.com/paper/2010.10732/) (2020)
- [The Effects of Regularization and Data Augmentation are Class Dependent](https://mlsanity.com/paper/2204.03632/) (2022)
- [Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification](https://mlsanity.com/paper/2004.09031/) (2020)
- [Wide Compression: Tensor Ring Nets](https://mlsanity.com/paper/1802.09052/) (2018)
- [Compressing Recurrent Neural Networks with Tensor Ring for Action Recognition](https://mlsanity.com/paper/1811.07503/) (2018)
- [Efficient Neural Network Training via Forward and Backward Propagation Sparsification](https://mlsanity.com/paper/2111.05685/) (2021)
- [Growing Efficient Deep Networks by Structured Continuous Sparsification](https://mlsanity.com/paper/2007.15353/) (2020)
- [Heuristic Rank Selection with Progressively Searching Tensor Ring Network](https://mlsanity.com/paper/2009.10580/) (2020)
- [Variational Bayesian Monte Carlo with Noisy Likelihoods](https://mlsanity.com/paper/2006.08655/) (2020)
- [HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks](https://mlsanity.com/paper/2301.09422/) (2023)
- [Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs](https://mlsanity.com/paper/2203.07808/) (2022)
- [Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs](https://mlsanity.com/paper/2008.09342/) (2020)

## Abstract (arXiv)

Deep neural networks typically impose significant computational loads and memory consumption. Moreover, the large parameters pose constraints on deploying the model on edge devices such as embedded systems. Tensor decomposition offers a clear advantage in compressing large-scale weight tensors. Nevertheless, direct utilization of low-rank decomposition typically leads to significant accuracy loss. This paper proposes a model compression method that integrates Variational Bayesian Matrix Factorization (VBMF) with orthogonal regularization. Initially, the model undergoes over-parameterization and training, with orthogonal regularization applied to enhance its likelihood of achieving the accuracy of the original model. Secondly, VBMF is employed to estimate the rank of the weight tensor at each layer. Our framework is sufficiently general to apply to other convolutional neural networks and easily adaptable to incorporate other tensor decomposition methods. Experimental results show that for both high and low compression ratios, our compression model exhibits advanced performance.