# Tensor Methods in Computer Vision and Deep Learning

arXiv: https://arxiv.org/abs/2107.03436 · HTML: https://mlsanity.com/paper/2107.03436/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 76 of its 240 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 30 of 76)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Tensor Decomposition for Signal Processing and Machine Learning](https://mlsanity.com/paper/1607.01668/) (2016)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://mlsanity.com/paper/1512.01274/) (2015)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Theoretical insights into the optimization landscape of over-parameterized shallow neural networks](https://mlsanity.com/paper/1707.04926/) (2017)
- [Tensor Ring Decomposition](https://mlsanity.com/paper/1606.05535/) (2016)
- [TensorLy: Tensor Learning in Python](https://mlsanity.com/paper/1610.09555/) (2016)
- [On the Power of Over-parametrization in Neural Networks with Quadratic Activation](https://mlsanity.com/paper/1803.01206/) (2018)
- [Shield: Fast, Practical Defense and Vaccination for Deep Learning using JPEG Compression](https://mlsanity.com/paper/1802.06816/) (2018)
- [Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network](https://mlsanity.com/paper/2008.05441/) (2020)
- [Deep Multi-task Representation Learning: A Tensor Factorisation Approach](https://mlsanity.com/paper/1605.06391/) (2016)
- [More is Less: A More Complicated Network with Less Inference Complexity](https://mlsanity.com/paper/1703.08651/) (2017)
- [Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination](https://mlsanity.com/paper/1401.6497/) (2014)
- [Deep Polynomial Neural Networks](https://mlsanity.com/paper/2006.13026/) (2020)

## Built on by (most cited first, 30 of 52)

- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs](https://mlsanity.com/paper/2310.00120/) (2023)
- [Guaranteed Tensor Recovery Fused Low-rankness and Smoothness](https://mlsanity.com/paper/2302.02155/) (2023)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [Bilinear MLPs enable weight-based mechanistic interpretability](https://mlsanity.com/paper/2410.08417/) (2024)
- [What is the Relationship between Tensor Factorizations and Circuits (and How Can We Exploit it)?](https://mlsanity.com/paper/2409.07953/) (2024)
- [Tensor Component Analysis for Interpreting the Latent Space of GANs](https://mlsanity.com/paper/2111.11736/) (2021)
- [Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling](https://mlsanity.com/paper/2205.14283/) (2022)
- [HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compression](https://mlsanity.com/paper/2211.16749/) (2022)
- [Guaranteed Approximation Bounds for Mixed-Precision Neural Operators](https://mlsanity.com/paper/2307.15034/) (2023)
- [Generative Learning of Continuous Data by Tensor Networks](https://mlsanity.com/paper/2310.20498/) (2023)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [tntorch: Tensor Network Learning with PyTorch](https://mlsanity.com/paper/2206.11128/) (2022)
- [TT-NF: Tensor Train Neural Fields](https://mlsanity.com/paper/2209.15529/) (2022)
- [Reinforcement Learning in Factored Action Spaces using Tensor Decompositions](https://mlsanity.com/paper/2110.14538/) (2021)
- [Quantization Aware Factorization for Deep Neural Network Compression](https://mlsanity.com/paper/2308.04595/) (2023)
- [TensorGRaD: Tensor Gradient Robust Decomposition for Memory-Efficient Neural Operator Training](https://mlsanity.com/paper/2501.02379/) (2025)
- [How to Train Unstable Looped Tensor Network](https://mlsanity.com/paper/2203.02617/) (2022)
- [Position: Tensor Networks are a Valuable Asset for Green AI](https://mlsanity.com/paper/2205.12961/) (2022)
- [Many-body Approximation for Non-negative Tensors](https://mlsanity.com/paper/2209.15338/) (2022)
- [Positive unlabeled learning with tensor networks](https://mlsanity.com/paper/2211.14085/) (2022)
- [MIGS: Multi-Identity Gaussian Splatting via Tensor Decomposition](https://mlsanity.com/paper/2407.07284/) (2024)
- [Boosting Defect Detection in Manufacturing using Tensor Convolutional Neural Networks](https://mlsanity.com/paper/2401.01373/) (2023)
- [FedBRB: An Effective Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learning](https://mlsanity.com/paper/2402.17202/) (2024)
- [Changing Base Without Losing Pace: A GPU-Efficient Alternative to MatMul in DNNs](https://mlsanity.com/paper/2503.12211/) (2025)
- [Multiscale Tensor Summation Factorization as a New Neural Network Layer (MTS Layer) for Multidimensional Data Processing](https://mlsanity.com/paper/2504.13975/) (2025)
- [Low-Rank Tensor Decompositions for the Theory of Neural Networks](https://mlsanity.com/paper/2508.18408/) (2025)
- [Approximation of Images via Generalized Higher Order Singular Value Decomposition over Finite-dimensional Commutative Semisimple Algebra](https://mlsanity.com/paper/2202.00450/) (2022)
- [A Comprehensive Review of Visual-Textual Sentiment Analysis from Social Media Networks](https://mlsanity.com/paper/2207.02160/) (2022)
- [Tensor Ring Optimized Quantum-Enhanced Tensor Neural Networks](https://mlsanity.com/paper/2310.01515/) (2023)

## Abstract (arXiv)

Tensors, or multidimensional arrays, are data structures that can naturally represent visual data of multiple dimensions. Inherently able to efficiently capture structured, latent semantic spaces and high-order interactions, tensors have a long history of applications in a wide span of computer vision problems. With the advent of the deep learning paradigm shift in computer vision, tensors have become even more fundamental. Indeed, essential ingredients in modern deep learning architectures, such as convolutions and attention mechanisms, can readily be considered as tensor mappings. In effect, tensor methods are increasingly finding significant applications in deep learning, including the design of memory and compute efficient network architectures, improving robustness to random noise and adversarial attacks, and aiding the theoretical understanding of deep networks. This article provides an in-depth and practical review of tensors and tensor methods in the context of representation learning and deep learning, with a particular focus on visual data analysis and computer vision applications. Concretely, besides fundamental work in tensor-based visual data analysis methods, we focus on recent developments that have brought on a gradual increase of tensor methods, especially in deep learning architectures, and their implications in computer vision applications. To further enable the newcomer to grasp such concepts quickly, we provide companion Python notebooks, covering key aspects of the paper and implementing them, step-by-step with TensorLy.