# Weight Matrix Dimensionality Reduction in Deep Learning via Kronecker Multi-layer Architectures

arXiv: https://arxiv.org/abs/2204.04273 · HTML: https://mlsanity.com/paper/2204.04273/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 8 Apr 2022.

## Builds on (most cited first, 9 of 9)

- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [The gap between theory and practice in function approximation with deep neural networks](https://mlsanity.com/paper/2001.07523/) (2020)
- [Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions](https://mlsanity.com/paper/2105.09513/) (2021)
- [Survey of Dropout Methods for Deep Neural Networks](https://mlsanity.com/paper/1904.13310/) (2019)
- [Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition](https://mlsanity.com/paper/2109.14710/) (2021)
- [Fast Convex Pruning of Deep Neural Networks](https://mlsanity.com/paper/1806.06457/) (2018)
- [Exploiting Local Structures with the Kronecker Layer in Convolutional Networks](https://mlsanity.com/paper/1512.09194/) (2015)

## Abstract (arXiv)

Deep learning using neural networks is an effective technique for generating models of complex data. However, training such models can be expensive when networks have large model capacity resulting from a large number of layers and nodes. For training in such a computationally prohibitive regime, dimensionality reduction techniques ease the computational burden, and allow implementations of more robust networks. We propose a novel type of such dimensionality reduction via a new deep learning architecture based on fast matrix multiplication of a Kronecker product decomposition; in particular our network construction can be viewed as a Kronecker product-induced sparsification of an "extended" fully connected network. Analysis and practical examples show that this architecture allows a neural network to be trained and implemented with a significant reduction in computational time and resources, while achieving a similar error level compared to a traditional feedforward neural network.