Low-Rank Tensor Decompositions for the Theory of Neural Networks
Not on the map. Why?
Abstract (arXiv)
The groundbreaking performance of deep neural networks (NNs) promoted a surge of interest in providing a mathematical basis to deep learning theory. Low-rank tensor decompositions are specially befitting for this task due to their close connection to NNs and their rich theoretical results. Different tensor decompositions have strong uniqueness guarantees, which allow for a direct interpretation of their factors, and polynomial time algorithms have been proposed to compute them. Through the connections between tensors and NNs, such results supported many important advances in the theory of NNs. In this review, we show how low-rank tensor methods--which have been a core tool in the signal processing and machine learning communities--play a fundamental role in theoretically explaining different aspects of the performance of deep NNs, including their expressivity, algorithmic learnability and computational hardness, generalization, and identifiability. Our goal is to give an accessible overview of existing approaches (developed by different communities, ranging from computer science to mathematics) in a coherent and unified way, and to open a broader perspective on the use of low-rank tensor decompositions for the theory of deep NNs.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 25 Aug 2025.
Builds on
32 of 54 references.
References · 32
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
- 2015Tensorizing Neural Networkscited by 250
- 2016Tensor Decomposition for Signal Processing and Machine Learningcited by 224
- 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
- 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
- 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 81
- 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
- 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 66
- 2019Expressive power of tensor-network factorizations for probabilistic modeling, with applications from hidden Markov models to quantum machine learningcited by 55
- 2021Tensor Methods in Computer Vision and Deep Learningcited by 52
- 2019Approximation spaces of deep neural networkscited by 51
20 more
- 2017The landscape of the spiked tensor modelcited by 34
- 2016Convolutional Rectifier Networks as Generalized Tensor Decompositionscited by 32
- 2017Expressive power of recurrent neural networkscited by 30
- 2021Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learningcited by 25
- 2017Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Designcited by 24
- 2018Learning Two-layer Neural Networks with Symmetric Inputscited by 22
- 2021Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activationscited by 15
- 2018On the Connection Between Learning Two-Layers Neural Networks and Tensor Decompositioncited by 14
- 2020Understanding Generalization in Deep Learning via Tensor Methodscited by 14
- 2023Tensor Decomposition for Model Reduction in Neural Networks: A Reviewcited by 14
- 2023Geometry-aware training of factorized layers in tensor Tucker formatcited by 12
- 2024LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Modelscited by 12
- 2025Identifiability of Deep Polynomial Neural Networkscited by 12
- 2019Robust and Resource Efficient Identification of Two Hidden Layer Neural Networkscited by 10
- 2014Score Function Features for Discriminative Learning: Matrix and Tensor Frameworkcited by 7
- 2020Grassmannian Optimization for Online Tensor Completion and Tracking with the t-SVDcited by 6
- 2020Approximation Theory of Tree Tensor Networks: Tensorized Univariate Functions -- Part Icited by 6
- 2023Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networkscited by 4
- 2016Identifiability of an X-rank decomposition of polynomial mapscited by 3
- 2024A Tensor Decomposition Perspective on Second-order RNNscited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.