# Tensorizing Neural Networks

arXiv: https://arxiv.org/abs/1509.06569 · HTML: https://mlsanity.com/paper/1509.06569/

## Key facts

- Cited by 250 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2015.

## Built on by (most cited first, 30 of 250)

- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges](https://mlsanity.com/paper/2201.08135/) (2022)
- [Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs](https://mlsanity.com/paper/2310.00120/) (2023)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models](https://mlsanity.com/paper/2101.11714/) (2021)
- [Tensor Methods in Computer Vision and Deep Learning](https://mlsanity.com/paper/2107.03436/) (2021)
- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [An end-to-end trainable hybrid classical-quantum classifier](https://mlsanity.com/paper/2102.02416/) (2021)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks](https://mlsanity.com/paper/2401.14109/) (2024)
- [Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models](https://mlsanity.com/paper/2203.01104/) (2022)
- [Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators](https://mlsanity.com/paper/2106.02205/) (2021)
- [Compute Better Spent: Replacing Dense Layers with Structured Matrices](https://mlsanity.com/paper/2406.06248/) (2024)
- [CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation](https://mlsanity.com/paper/2502.10940/) (2025)
- [QTN-VQC: An End-to-End Learning framework for Quantum Neural Networks](https://mlsanity.com/paper/2110.03861/) (2021)
- [Theoretical Error Performance Analysis for Variational Quantum Circuit Based Functional Regression](https://mlsanity.com/paper/2206.04804/) (2022)
- [AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuning](https://mlsanity.com/paper/2406.18060/) (2024)
- [Spatio-Temporal Pruning and Quantization for Low-latency Spiking Neural Networks](https://mlsanity.com/paper/2104.12528/) (2021)
- [HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks](https://mlsanity.com/paper/2301.09422/) (2023)
- [Double Similarity Distillation for Semantic Image Segmentation](https://mlsanity.com/paper/2107.08591/) (2021)
- [Local approximate Gaussian process regression for data-driven constitutive laws: Development and comparison with neural networks](https://mlsanity.com/paper/2105.04554/) (2021)
- [Tensor networks for quantum machine learning](https://mlsanity.com/paper/2303.11735/) (2023)
- [Exploring Extreme Parameter Compression for Pre-trained Language Models](https://mlsanity.com/paper/2205.10036/) (2022)
- [CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization](https://mlsanity.com/paper/2405.14377/) (2024)
- [Layer Folding: Neural Network Depth Reduction using Activation Linearization](https://mlsanity.com/paper/2106.09309/) (2021)
- [Large-Scale Learning with Fourier Features and Tensor Decompositions](https://mlsanity.com/paper/2109.01545/) (2021)
- [Tensor Decomposition for Model Reduction in Neural Networks: A Review](https://mlsanity.com/paper/2304.13539/) (2023)
- [Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs](https://mlsanity.com/paper/2408.01008/) (2024)

## Abstract (arXiv)

Deep neural networks currently demonstrate state-of-the-art performance in several domains. At the same time, models of this class are very demanding in terms of computational resources. In particular, a large amount of memory is required by commonly used fully-connected layers, making it hard to use the models on low-end devices and stopping the further increase of the model size. In this paper we convert the dense weight matrices of the fully-connected layers to the Tensor Train format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved. In particular, for the Very Deep VGG networks we report the compression factor of the dense weight matrix of a fully-connected layer up to 200000 times leading to the compression factor of the whole network up to 7 times.