# Low-Rank Tensor Decompositions for the Theory of Neural Networks

arXiv: https://arxiv.org/abs/2508.18408 · HTML: https://mlsanity.com/paper/2508.18408/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 54 references that are in the map.
- First posted on arXiv 25 Aug 2025.

## Builds on (most cited first, 30 of 32)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Tensor Decomposition for Signal Processing and Machine Learning](https://mlsanity.com/paper/1607.01668/) (2016)
- [On the Expressive Power of Deep Learning: A Tensor Analysis](https://mlsanity.com/paper/1509.05009/) (2015)
- [Recovery Guarantees for One-hidden-layer Neural Networks](https://mlsanity.com/paper/1706.03175/) (2017)
- [Learning One-hidden-layer Neural Networks with Landscape Design](https://mlsanity.com/paper/1711.00501/) (2017)
- [Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network](https://mlsanity.com/paper/2008.05441/) (2020)
- [On the Expressive Power of Deep Polynomial Neural Networks](https://mlsanity.com/paper/1905.12207/) (2019)
- [Expressive power of tensor-network factorizations for probabilistic modeling, with applications from hidden Markov models to quantum machine learning](https://mlsanity.com/paper/1907.03741/) (2019)
- [Tensor Methods in Computer Vision and Deep Learning](https://mlsanity.com/paper/2107.03436/) (2021)
- [Approximation spaces of deep neural networks](https://mlsanity.com/paper/1905.01208/) (2019)
- [The landscape of the spiked tensor model](https://mlsanity.com/paper/1711.05424/) (2017)
- [Convolutional Rectifier Networks as Generalized Tensor Decompositions](https://mlsanity.com/paper/1603.00162/) (2016)
- [Expressive power of recurrent neural networks](https://mlsanity.com/paper/1711.00811/) (2017)
- [Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00136/) (2021)
- [Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design](https://mlsanity.com/paper/1704.01552/) (2017)
- [Learning Two-layer Neural Networks with Symmetric Inputs](https://mlsanity.com/paper/1810.06793/) (2018)
- [Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activations](https://mlsanity.com/paper/2107.10209/) (2021)
- [On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition](https://mlsanity.com/paper/1802.07301/) (2018)
- [Understanding Generalization in Deep Learning via Tensor Methods](https://mlsanity.com/paper/2001.05070/) (2020)
- [Tensor Decomposition for Model Reduction in Neural Networks: A Review](https://mlsanity.com/paper/2304.13539/) (2023)
- [Geometry-aware training of factorized layers in tensor Tucker format](https://mlsanity.com/paper/2305.19059/) (2023)
- [LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2402.11417/) (2024)
- [Identifiability of Deep Polynomial Neural Networks](https://mlsanity.com/paper/2506.17093/) (2025)
- [Robust and Resource Efficient Identification of Two Hidden Layer Neural Networks](https://mlsanity.com/paper/1907.00485/) (2019)
- [Score Function Features for Discriminative Learning: Matrix and Tensor Framework](https://mlsanity.com/paper/1412.2863/) (2014)
- [Grassmannian Optimization for Online Tensor Completion and Tracking with the t-SVD](https://mlsanity.com/paper/2001.11419/) (2020)
- [Approximation Theory of Tree Tensor Networks: Tensorized Univariate Functions -- Part I](https://mlsanity.com/paper/2007.00118/) (2020)
- [Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networks](https://mlsanity.com/paper/2303.00196/) (2023)

## Built on by (most cited first, 2 of 2)

- [Noise-robust Contrastive Learning for Critical Transition Detection in Dynamical Systems](https://mlsanity.com/paper/2512.12523/) (2025)
- [When AI meets quantum information: A comprehensive review](https://mlsanity.com/paper/2607.00365/) (2026)

## Abstract (arXiv)

The groundbreaking performance of deep neural networks (NNs) promoted a surge of interest in providing a mathematical basis to deep learning theory. Low-rank tensor decompositions are specially befitting for this task due to their close connection to NNs and their rich theoretical results. Different tensor decompositions have strong uniqueness guarantees, which allow for a direct interpretation of their factors, and polynomial time algorithms have been proposed to compute them. Through the connections between tensors and NNs, such results supported many important advances in the theory of NNs. In this review, we show how low-rank tensor methods--which have been a core tool in the signal processing and machine learning communities--play a fundamental role in theoretically explaining different aspects of the performance of deep NNs, including their expressivity, algorithmic learnability and computational hardness, generalization, and identifiability. Our goal is to give an accessible overview of existing approaches (developed by different communities, ranging from computer science to mathematics) in a coherent and unified way, and to open a broader perspective on the use of low-rank tensor decompositions for the theory of deep NNs.