# Compressing deep neural networks by matrix product operators

arXiv: https://arxiv.org/abs/1904.06194 · HTML: https://mlsanity.com/paper/1904.06194/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 28 of 28)

- [Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study](https://mlsanity.com/paper/2307.08072/) (2023)
- [Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models](https://mlsanity.com/paper/2203.01104/) (2022)
- [Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators](https://mlsanity.com/paper/2106.02205/) (2021)
- [Tensor networks for interpretable and efficient quantum-inspired machine learning](https://mlsanity.com/paper/2311.11258/) (2023)
- [Exploring Extreme Parameter Compression for Pre-trained Language Models](https://mlsanity.com/paper/2205.10036/) (2022)
- [Tensor networks and efficient descriptions of classical data](https://mlsanity.com/paper/2103.06872/) (2021)
- [A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks](https://mlsanity.com/paper/2205.15307/) (2022)
- [Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression](https://mlsanity.com/paper/2405.12591/) (2024)
- [Quantum-Enhanced LLM Efficient Fine Tuning](https://mlsanity.com/paper/2503.12790/) (2025)
- [Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation](https://mlsanity.com/paper/2411.06448/) (2024)
- [Sample Selection via Contrastive Fragmentation for Noisy Label Regression](https://mlsanity.com/paper/2502.17771/) (2025)
- [Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks](https://mlsanity.com/paper/2505.20132/) (2025)
- [DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models](https://mlsanity.com/paper/2412.20891/) (2024)
- [Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT](https://mlsanity.com/paper/2603.28534/) (2026)
- [Quantum-inspired tensor network for Earth science](https://mlsanity.com/paper/2301.07528/) (2023)
- [An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning](https://mlsanity.com/paper/2211.16703/) (2022)
- [Density peak clustering using tensor network](https://mlsanity.com/paper/2302.00192/) (2023)
- [Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture](https://mlsanity.com/paper/2303.16753/) (2023)
- [Less is More! A slim architecture for optimal language translation](https://mlsanity.com/paper/2305.10991/) (2023)
- [Efficient Finite Initialization with Partial Norms for Tensorized Neural Networks and Tensor Networks Algorithms](https://mlsanity.com/paper/2309.06577/) (2023)
- [A quatum inspired neural network for geometric modeling](https://mlsanity.com/paper/2401.01801/) (2024)
- [Universal replication of chaotic characteristics by classical and quantum machine learning](https://mlsanity.com/paper/2405.08484/) (2024)
- [Quantum-inspired tensor networks in machine learning models](https://mlsanity.com/paper/2604.14287/) (2026)
- [Strategic Over-Parameterization for Generalizable Low-Rank Adaptation](https://mlsanity.com/paper/2605.16470/) (2026)
- [Fast Tensorization of Neural Networks via Slice-wise Feature Distillation](https://mlsanity.com/paper/2605.19842/) (2026)
- [A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Models](https://mlsanity.com/paper/2605.25344/) (2026)
- [Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scale](https://mlsanity.com/paper/2609.34292/) (2026)
- [Gaussian Universality and Its Breakdown in Tensor-Network Machine Learning](https://mlsanity.com/paper/2610.06080/) (2026)

## Abstract (arXiv)

A deep neural network is a parametrization of a multilayer mapping of signals in terms of many alternatively arranged linear and nonlinear transformations. The linear transformations, which are generally used in the fully connected as well as convolutional layers, contain most of the variational parameters that are trained and stored. Compressing a deep neural network to reduce its number of variational parameters but not its prediction power is an important but challenging problem toward the establishment of an optimized scheme in training efficiently these parameters and in lowering the risk of overfitting. Here we show that this problem can be effectively solved by representing linear transformations with matrix product operators (MPOs), which is a tensor network originally proposed in physics to characterize the short-range entanglement in one-dimensional quantum states. We have tested this approach in five typical neural networks, including FC2, LeNet-5, VGG, ResNet, and DenseNet on two widely used data sets, namely, MNIST and CIFAR-10, and found that this MPO representation indeed sets up a faithful and efficient mapping between input and output signals, which can keep or even improve the prediction accuracy with a dramatically reduced number of parameters. Our method greatly simplifies the representations in deep learning, and opens a possible route toward establishing a framework of modern neural networks which might be simpler and cheaper, but more efficient.