# Compression of Fully-Connected Layer in Neural Network by Kronecker Product

arXiv: https://arxiv.org/abs/1507.05775 · HTML: https://mlsanity.com/paper/1507.05775/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2015.

## Built on by (most cited first, 9 of 9)

- [Kronecker Decomposition for GPT Compression](https://mlsanity.com/paper/2110.08152/) (2021)
- [KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation](https://mlsanity.com/paper/2109.06243/) (2021)
- [TQCompressor: improving tensor decomposition methods in neural networks via permutations](https://mlsanity.com/paper/2401.16367/) (2024)
- [Kronecker Decomposition for Knowledge Graph Embeddings](https://mlsanity.com/paper/2205.06560/) (2022)
- [Learning Robust and Lightweight Model through Separable Structured Transformations](https://mlsanity.com/paper/2112.13551/) (2021)
- [Fully invertible hyperbolic neural networks for segmenting large-scale surface and sub-surface data](https://mlsanity.com/paper/2407.00595/) (2024)
- [Layer-Specific Optimization: Sensitivity Based Convolution Layers Basis Search](https://mlsanity.com/paper/2408.06024/) (2024)
- [Krony-PT: GPT2 compressed with Kronecker Products](https://mlsanity.com/paper/2412.12351/) (2024)
- [A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Models](https://mlsanity.com/paper/2605.25344/) (2026)

## Abstract (arXiv)

In this paper we propose and study a technique to reduce the number of parameters and computation time in fully-connected layers of neural networks using Kronecker product, at a mild cost of the prediction quality. The technique proceeds by replacing Fully-Connected layers with so-called Kronecker Fully-Connected layers, where the weight matrices of the FC layers are approximated by linear combinations of multiple Kronecker products of smaller matrices. In particular, given a model trained on SVHN dataset, we are able to construct a new KFC model with 73% reduction in total number of parameters, while the error only rises mildly. In contrast, using low-rank method can only achieve 35% reduction in total number of parameters given similar quality degradation allowance. If we only compare the KFC layer with its counterpart fully-connected layer, the reduction in the number of parameters exceeds 99%. The amount of computation is also reduced as we replace matrix product of the large matrices in FC layers with matrix products of a few smaller matrices in KFC layers. Further experiments on MNIST, SVHN and some Chinese Character recognition models also demonstrate effectiveness of our technique.