# Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition

arXiv: https://arxiv.org/abs/1412.6553 · HTML: https://mlsanity.com/paper/1412.6553/

## Key facts

- Cited by 203 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2014.

## Built on by (most cited first, 30 of 203)

- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models](https://mlsanity.com/paper/2312.05821/) (2023)
- [FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2108.06098/) (2021)
- [FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer](https://mlsanity.com/paper/2212.03145/) (2022)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs](https://mlsanity.com/paper/2310.00120/) (2023)
- [Tensor Methods in Computer Vision and Deep Learning](https://mlsanity.com/paper/2107.03436/) (2021)
- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Manifold Regularized Dynamic Network Pruning](https://mlsanity.com/paper/2103.05861/) (2021)
- [Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations](https://mlsanity.com/paper/2205.13571/) (2022)
- [The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks](https://mlsanity.com/paper/2203.04466/) (2022)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [Towards Compact CNNs via Collaborative Compression](https://mlsanity.com/paper/2105.11228/) (2021)
- [Revisiting Dynamic Convolution via Matrix Decomposition](https://mlsanity.com/paper/2103.08756/) (2021)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization](https://mlsanity.com/paper/2111.14655/) (2021)
- [Content-Aware GAN Compression](https://mlsanity.com/paper/2104.02244/) (2021)
- [CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation](https://mlsanity.com/paper/2502.10940/) (2025)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [Data-Efficient Structured Pruning via Submodular Optimization](https://mlsanity.com/paper/2203.04940/) (2022)
- [Bit-Mixer: Mixed-precision networks with runtime bit-width selection](https://mlsanity.com/paper/2103.17267/) (2021)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimization](https://mlsanity.com/paper/2110.14807/) (2021)
- [Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward](https://mlsanity.com/paper/2402.01799/) (2024)
- [Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions](https://mlsanity.com/paper/2209.08554/) (2022)
- [HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks](https://mlsanity.com/paper/2301.09422/) (2023)
- [Scaling Up Exact Neural Network Compression by ReLU Stability](https://mlsanity.com/paper/2102.07804/) (2021)

## Abstract (arXiv)

We propose a simple two-step approach for speeding up convolution layers within large convolutional neural networks based on tensor decomposition and discriminative fine-tuning. Given a layer, we use non-linear least squares to compute a low-rank CP-decomposition of the 4D convolution kernel tensor into a sum of a small number of rank-one tensors. At the second step, this decomposition is used to replace the original convolutional layer with a sequence of four convolutional layers with small kernels. After such replacement, the entire network is fine-tuned on the training data using standard backpropagation process. We evaluate this approach on two CNNs and show that it is competitive with previous approaches, leading to higher obtained CPU speedups at the cost of lower accuracy drops for the smaller of the two networks. Thus, for the 36-class character classification CNN, our approach obtains a 8.5x CPU speedup of the whole network with only minor accuracy drop (1% from 91% to 90%). For the standard ImageNet architecture (AlexNet), the approach speeds up the second convolution layer by a factor of 4x at the cost of 1% increase of the overall top-5 classification error.