# Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition

arXiv: https://arxiv.org/abs/2109.14710 · HTML: https://mlsanity.com/paper/2109.14710/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 29 Sep 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Trained Ternary Quantization](https://mlsanity.com/paper/1612.01064/) (2016)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Learning the Number of Neurons in Deep Networks](https://mlsanity.com/paper/1611.06321/) (2016)
- [Ultimate tensorization: compressing convolutional and FC layers alike](https://mlsanity.com/paper/1611.03214/) (2016)
- [Wide Compression: Tensor Ring Nets](https://mlsanity.com/paper/1802.09052/) (2018)
- [Knowledge Distillation with Adversarial Samples Supporting Decision Boundary](https://mlsanity.com/paper/1805.05532/) (2018)
- [Compression-aware Training of Deep Networks](https://mlsanity.com/paper/1711.02638/) (2017)
- [Efficient Low Rank Tensor Ring Completion](https://mlsanity.com/paper/1707.08184/) (2017)
- [OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networks](https://mlsanity.com/paper/1905.11664/) (2019)

## Built on by (most cited first, 14 of 14)

- [KronA: Parameter Efficient Tuning with Kronecker Adapter](https://mlsanity.com/paper/2212.10650/) (2022)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized Diffusion Models](https://mlsanity.com/paper/2402.17412/) (2024)
- [Building Variable-sized Models via Learngene Pool](https://mlsanity.com/paper/2312.05743/) (2023)
- [Activation Map Compression through Tensor Decomposition for Deep Learning](https://mlsanity.com/paper/2411.06346/) (2024)
- [Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training](https://mlsanity.com/paper/2303.13635/) (2023)
- [CAMEx: Curvature-aware Merging of Experts](https://mlsanity.com/paper/2502.18821/) (2025)
- [Understanding MLP-Mixer as a Wide and Sparse MLP](https://mlsanity.com/paper/2306.01470/) (2023)
- [An Efficient Training Algorithm for Models with Block-wise Sparsity](https://mlsanity.com/paper/2503.21928/) (2025)
- [Dynamic Rank Adjustment for Accurate and Efficient Neural Network Training](https://mlsanity.com/paper/2508.08625/) (2025)
- [Weight Matrix Dimensionality Reduction in Deep Learning via Kronecker Multi-layer Architectures](https://mlsanity.com/paper/2204.04273/) (2022)
- [SeKron: A Decomposition Method Supporting Many Factorization Structures](https://mlsanity.com/paper/2210.06299/) (2022)
- [A priori compression of convolutional neural networks for wave simulators](https://mlsanity.com/paper/2304.04964/) (2023)
- [Efficient Resource-Constrained Training of Transformers via Subspace Optimization](https://mlsanity.com/paper/2510.09160/) (2025)

## Abstract (arXiv)

Modern Convolutional Neural Network (CNN) architectures, despite their superiority in solving various problems, are generally too large to be deployed on resource constrained edge devices. In this paper, we reduce memory usage and floating-point operations required by convolutional layers in CNNs. We compress these layers by generalizing the Kronecker Product Decomposition to apply to multidimensional tensors, leading to the Generalized Kronecker Product Decomposition (GKPD). Our approach yields a plug-and-play module that can be used as a drop-in replacement for any convolutional layer. Experimental results for image classification on CIFAR-10 and ImageNet datasets using ResNet, MobileNetv2 and SeNet architectures substantiate the effectiveness of our proposed approach. We find that GKPD outperforms state-of-the-art decomposition methods including Tensor-Train and Tensor-Ring as well as other relevant compression methods such as pruning and knowledge distillation.