# Ultimate tensorization: compressing convolutional and FC layers alike

arXiv: https://arxiv.org/abs/1611.03214 · HTML: https://mlsanity.com/paper/1611.03214/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2016.

## Built on by (most cited first, 30 of 56)

- [Task Singular Vectors: Reducing Task Interference in Model Merging](https://mlsanity.com/paper/2412.00081/) (2024)
- [Tensor Methods in Computer Vision and Deep Learning](https://mlsanity.com/paper/2107.03436/) (2021)
- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models](https://mlsanity.com/paper/2203.01104/) (2022)
- [Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators](https://mlsanity.com/paper/2106.02205/) (2021)
- [Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization](https://mlsanity.com/paper/2402.12550/) (2024)
- [QTN-VQC: An End-to-End Learning framework for Quantum Neural Networks](https://mlsanity.com/paper/2110.03861/) (2021)
- [Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition](https://mlsanity.com/paper/2109.14710/) (2021)
- [Tensor Decomposition for Model Reduction in Neural Networks: A Review](https://mlsanity.com/paper/2304.13539/) (2023)
- [Joint Matrix Decomposition for Deep Convolutional Neural Networks Compression](https://mlsanity.com/paper/2107.04386/) (2021)
- [Exploiting Low-Rank Tensor-Train Deep Neural Networks Based on Riemannian Gradient Descent With Illustrations of Speech Processing](https://mlsanity.com/paper/2203.06031/) (2022)
- [Quantum-inspired event reconstruction with Tensor Networks: Matrix Product States](https://mlsanity.com/paper/2106.08334/) (2021)
- [Towards Practical Control of Singular Values of Convolutional Layers](https://mlsanity.com/paper/2211.13771/) (2022)
- [A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks](https://mlsanity.com/paper/2205.15307/) (2022)
- [More Efficient Sampling for Tensor Decomposition With Worst-Case Guarantees](https://mlsanity.com/paper/2110.07631/) (2021)
- [Classical versus Quantum: comparing Tensor Network-based Quantum Circuits on LHC data](https://mlsanity.com/paper/2202.10471/) (2022)
- [TT-NF: Tensor Train Neural Fields](https://mlsanity.com/paper/2209.15529/) (2022)
- [On the Opportunities of Green Computing: A Survey](https://mlsanity.com/paper/2311.00447/) (2023)
- [Spectral Tensor Train Parameterization of Deep Learning Layers](https://mlsanity.com/paper/2103.04217/) (2021)
- [Rank-R FNN: A Tensor-Based Learning Model for High-Order Data Classification](https://mlsanity.com/paper/2104.05048/) (2021)
- [Update Compression for Deep Neural Networks on the Edge](https://mlsanity.com/paper/2203.04516/) (2022)
- [Nimble GNN Embedding with Tensor-Train Decomposition](https://mlsanity.com/paper/2206.10581/) (2022)
- [LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing](https://mlsanity.com/paper/2505.21732/) (2025)
- [Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training](https://mlsanity.com/paper/2303.13635/) (2023)
- [Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation](https://mlsanity.com/paper/2411.06448/) (2024)
- [LightLayers: Parameter Efficient Dense and Convolutional Layers for Image Classification](https://mlsanity.com/paper/2101.02268/) (2021)
- [FAMLP: A Frequency-Aware MLP-Like Architecture For Domain Generalization](https://mlsanity.com/paper/2203.12893/) (2022)
- [MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2506.09105/) (2025)

## Abstract (arXiv)

Convolutional neural networks excel in image recognition tasks, but this comes at the cost of high computational and memory complexity. To tackle this problem, [1] developed a tensor factorization framework to compress fully-connected layers. In this paper, we focus on compressing convolutional layers. We show that while the direct application of the tensor framework [1] to the 4-dimensional kernel of convolution does compress the layer, we can do better. We reshape the convolutional kernel into a tensor of higher order and factorize it. We combine the proposed approach with the previous work to compress both convolutional and fully-connected layers of a network and achieve 80x network compression rate with 1.1% accuracy drop on the CIFAR-10 dataset.