# Joint Matrix Decomposition for Deep Convolutional Neural Networks Compression

arXiv: https://arxiv.org/abs/2107.04386 · HTML: https://mlsanity.com/paper/2107.04386/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015)
- [Deep Learning for Audio Signal Processing](https://mlsanity.com/paper/1905.00078/) (2019)
- [Knowledge Transfer with Jacobian Matching](https://mlsanity.com/paper/1803.00443/) (2018)
- [Ultimate tensorization: compressing convolutional and FC layers alike](https://mlsanity.com/paper/1611.03214/) (2016)

## Built on by (most cited first, 10 of 10)

- [WHC: Weighted Hybrid Criterion for Filter Pruning on Convolutional Neural Networks](https://mlsanity.com/paper/2302.08185/) (2023)
- [POCKET: Pruning Random Convolution Kernels for Time Series Classification from a Feature Selection Perspective](https://mlsanity.com/paper/2309.08499/) (2023)
- [Edge AI: Evaluation of Model Compression Techniques for Convolutional Neural Networks](https://mlsanity.com/paper/2409.02134/) (2024)
- [TropNNC: Structured Neural Network Compression Using Tropical Geometry](https://mlsanity.com/paper/2409.03945/) (2024)
- [CAST: Compositional Analysis via Spectral Tracking for Understanding Transformer Layer Functions](https://mlsanity.com/paper/2510.14262/) (2025)
- [Sub-network Multi-objective Evolutionary Algorithm for Filter Pruning](https://mlsanity.com/paper/2211.01957/) (2022)
- [Layer-Specific Optimization: Sensitivity Based Convolution Layers Basis Search](https://mlsanity.com/paper/2408.06024/) (2024)
- [P2U: Progressive Precision Update For Efficient Model Distribution](https://mlsanity.com/paper/2506.22871/) (2025)
- [Efficient CNN Compression via Multi-method Low Rank Factorization and Feature Map Similarity](https://mlsanity.com/paper/2510.00062/) (2025)
- [Fast Tensorization of Neural Networks via Slice-wise Feature Distillation](https://mlsanity.com/paper/2605.19842/) (2026)

## Abstract (arXiv)

Deep convolutional neural networks (CNNs) with a large number of parameters require intensive computational resources, and thus are hard to be deployed in resource-constrained platforms. Decomposition-based methods, therefore, have been utilized to compress CNNs in recent years. However, since the compression factor and performance are negatively correlated, the state-of-the-art works either suffer from severe performance degradation or have relatively low compression factors. To overcome this problem, we propose to compress CNNs and alleviate performance degradation via joint matrix decomposition, which is different from existing works that compressed layers separately. The idea is inspired by the fact that there are lots of repeated modules in CNNs. By projecting weights with the same structures into the same subspace, networks can be jointly compressed with larger ranks. In particular, three joint matrix decomposition schemes are developed, and the corresponding optimization approaches based on Singular Value Decomposition are proposed. Extensive experiments are conducted across three challenging compact CNNs for different benchmark data sets to demonstrate the superior performance of our proposed algorithms. As a result, our methods can compress the size of ResNet-34 by 22X with slighter accuracy degradation compared with several state-of-the-art methods.