# Tensor Regression Networks

arXiv: https://arxiv.org/abs/1707.08308 · HTML: https://mlsanity.com/paper/1707.08308/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2017.

## Built on by (most cited first, 30 of 44)

- [Initialization and Regularization of Factorized Neural Layers](https://mlsanity.com/paper/2105.01029/) (2021)
- [Tensor Methods in Computer Vision and Deep Learning](https://mlsanity.com/paper/2107.03436/) (2021)
- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [Tensor Component Analysis for Interpreting the Latent Space of GANs](https://mlsanity.com/paper/2111.11736/) (2021)
- [Permutation Search of Tensor Network Structures via Local Sampling](https://mlsanity.com/paper/2206.06597/) (2022)
- [Alternating Local Enumeration (TnALE): Solving Tensor Network Structure Search with Fewer Evaluations](https://mlsanity.com/paper/2304.12875/) (2023)
- [Tensor-view Topological Graph Neural Network](https://mlsanity.com/paper/2401.12007/) (2024)
- [HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compression](https://mlsanity.com/paper/2211.16749/) (2022)
- [Tensor Decomposition Based Attention Module for Spiking Neural Networks](https://mlsanity.com/paper/2310.14576/) (2023)
- [Factor Augmented Tensor-on-Tensor Neural Networks](https://mlsanity.com/paper/2405.19610/) (2024)
- [Rank-R FNN: A Tensor-Based Learning Model for High-Order Data Classification](https://mlsanity.com/paper/2104.05048/) (2021)
- [tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models (LLMs)](https://mlsanity.com/paper/2402.02456/) (2024)
- [Language Modeling Using Tensor Trains](https://mlsanity.com/paper/2405.04590/) (2024)
- [Computational and Statistical Guarantees for Tensor-on-Tensor Regression with Tensor Train Decomposition](https://mlsanity.com/paper/2406.06002/) (2024)
- [Semi-tensor Product-based TensorDecomposition for Neural Network Compression](https://mlsanity.com/paper/2109.15200/) (2021)
- [MIGS: Multi-Identity Gaussian Splatting via Tensor Decomposition](https://mlsanity.com/paper/2407.07284/) (2024)
- [Conditional Prediction ROC Bands for Graph Classification](https://mlsanity.com/paper/2410.15239/) (2024)
- [Conditional Uncertainty Quantification for Tensorized Topological Neural Networks](https://mlsanity.com/paper/2410.15241/) (2024)
- [Tensor-Fused Multi-View Graph Contrastive Learning](https://mlsanity.com/paper/2410.15247/) (2024)
- [Patch Tracking-based Streaming Tensor Ring Completion for Visual Data Recovery](https://mlsanity.com/paper/2105.14620/) (2021)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [Causal Deep Learning](https://mlsanity.com/paper/2301.00314/) (2023)
- [Tensor-on-tensor Regression Neural Networks for Process Modeling with High-dimensional Data](https://mlsanity.com/paper/2510.05329/) (2025)
- [High-Dimensional Uncertainty Quantification via Tensor Regression with Rank Determination and Adaptive Sampling](https://mlsanity.com/paper/2103.17236/) (2021)
- [A New Measure of Model Redundancy for Compressed Convolutional Neural Networks](https://mlsanity.com/paper/2112.04857/) (2021)
- [Brain Structural Saliency Over The Ages](https://mlsanity.com/paper/2202.11690/) (2022)
- [conv_einsum: A Framework for Representation and Fast Evaluation of Multilinear Operations in Convolutional Tensorial Neural Networks](https://mlsanity.com/paper/2401.03384/) (2024)
- [TEAFormers: TEnsor-Augmented Transformers for Multi-Dimensional Time Series Forecasting](https://mlsanity.com/paper/2410.20439/) (2024)
- [Efficient Tensor Contraction via Fast Count Sketch](https://mlsanity.com/paper/2106.13062/) (2021)
- [Dictionary-based Low-Rank Approximations and the Mixed Sparse Coding problem](https://mlsanity.com/paper/2111.12399/) (2021)

## Abstract (arXiv)

Convolutional neural networks typically consist of many convolutional layers followed by one or more fully connected layers. While convolutional layers map between high-order activation tensors, the fully connected layers operate on flattened activation vectors. Despite empirical success, this approach has notable drawbacks. Flattening followed by fully connected layers discards multilinear structure in the activations and requires many parameters. We address these problems by incorporating tensor algebraic operations that preserve multilinear structure at every layer. First, we introduce Tensor Contraction Layers (TCLs) that reduce the dimensionality of their input while preserving their multilinear structure using tensor contraction. Next, we introduce Tensor Regression Layers (TRLs), which express outputs through a low-rank multilinear mapping from a high-order activation tensor to an output tensor of arbitrary order. We learn the contraction and regression factors end-to-end, and produce accurate nets with fewer parameters. Additionally, our layers regularize networks by imposing low-rank constraints on the activations (TCL) and regression weights (TRL). Experiments on ImageNet show that, applied to VGG and ResNet architectures, TCLs and TRLs reduce the number of parameters compared to fully connected layers by more than 65% while maintaining or increasing accuracy. In addition to the space savings, our approach's ability to leverage topological structure can be crucial for structured data such as MRI. In particular, we demonstrate significant performance improvements over comparable architectures on three tasks associated with the UK Biobank dataset.