# Tensor Contraction Layers for Parsimonious Deep Nets

arXiv: https://arxiv.org/abs/1706.00439 · HTML: https://mlsanity.com/paper/1706.00439/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2017.

## Built on by (most cited first, 16 of 16)

- [Tensor Methods in Computer Vision and Deep Learning](https://mlsanity.com/paper/2107.03436/) (2021)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [Tensor-view Topological Graph Neural Network](https://mlsanity.com/paper/2401.12007/) (2024)
- [TRAWL: Tensor Reduced and Approximated Weights for Large Language Models](https://mlsanity.com/paper/2406.17261/) (2024)
- [MIGS: Multi-Identity Gaussian Splatting via Tensor Decomposition](https://mlsanity.com/paper/2407.07284/) (2024)
- [Conditional Uncertainty Quantification for Tensorized Topological Neural Networks](https://mlsanity.com/paper/2410.15241/) (2024)
- [Tensor-Fused Multi-View Graph Contrastive Learning](https://mlsanity.com/paper/2410.15247/) (2024)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [A New Measure of Model Redundancy for Compressed Convolutional Neural Networks](https://mlsanity.com/paper/2112.04857/) (2021)
- [Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey](https://mlsanity.com/paper/2208.10498/) (2022)
- [Tensor Ring Optimized Quantum-Enhanced Tensor Neural Networks](https://mlsanity.com/paper/2310.01515/) (2023)
- [conv_einsum: A Framework for Representation and Fast Evaluation of Multilinear Operations in Convolutional Tensorial Neural Networks](https://mlsanity.com/paper/2401.03384/) (2024)
- [TEAFormers: TEnsor-Augmented Transformers for Multi-Dimensional Time Series Forecasting](https://mlsanity.com/paper/2410.20439/) (2024)
- [Bayesian Deep Learning Hyperparameter Search for Robust Function Mapping to Polynomials with Noise](https://mlsanity.com/paper/2106.12532/) (2021)
- [Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning](https://mlsanity.com/paper/2401.08014/) (2024)
- [FLAASH: Flexible Accelerator Architecture for Sparse High-Order Tensor Contraction](https://mlsanity.com/paper/2404.16317/) (2024)

## Abstract (arXiv)

Tensors offer a natural representation for many kinds of data frequently encountered in machine learning. Images, for example, are naturally represented as third order tensors, where the modes correspond to height, width, and channels. Tensor methods are noted for their ability to discover multi-dimensional dependencies, and tensor decompositions in particular, have been used to produce compact low-rank approximations of data. In this paper, we explore the use of tensor contractions as neural network layers and investigate several ways to apply them to activation tensors. Specifically, we propose the Tensor Contraction Layer (TCL), the first attempt to incorporate tensor contractions as end-to-end trainable neural network layers. Applied to existing networks, TCLs reduce the dimensionality of the activation tensors and thus the number of model parameters. We evaluate the TCL on the task of image recognition, augmenting two popular networks (AlexNet, VGG). The resulting models are trainable end-to-end. Applying the TCL to the task of image recognition, using the CIFAR100 and ImageNet datasets, we evaluate the effect of parameter reduction via tensor contraction on performance. We demonstrate significant model compression without significant impact on the accuracy and, in some cases, improved performance.