# Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositions

arXiv: https://arxiv.org/abs/1705.02302 · HTML: https://mlsanity.com/paper/1705.02302/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2017.

## Built on by (most cited first, 13 of 13)

- [Implicit Regularization in Tensor Factorization](https://mlsanity.com/paper/2102.09972/) (2021)
- [The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design](https://mlsanity.com/paper/2110.04541/) (2021)
- [Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks](https://mlsanity.com/paper/2201.11729/) (2022)
- [Learning a Self-Expressive Network for Subspace Clustering](https://mlsanity.com/paper/2110.04318/) (2021)
- [Which transformer architecture fits my data? A vocabulary bottleneck in self-attention](https://mlsanity.com/paper/2105.03928/) (2021)
- [On the Ability of Graph Neural Networks to Model Interactions Between Vertices](https://mlsanity.com/paper/2211.16494/) (2022)
- [Towards Extremely Compact RNNs for Video Recognition with Fully Decomposed Hierarchical Tucker Structure](https://mlsanity.com/paper/2104.05758/) (2021)
- [What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglement](https://mlsanity.com/paper/2303.11249/) (2023)
- [Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning](https://mlsanity.com/paper/2408.13767/) (2024)
- [STD-NET: Search of Image Steganalytic Deep-learning Architecture via Hierarchical Tensor Decomposition](https://mlsanity.com/paper/2206.05651/) (2022)
- [Transformer Vs. MLP-Mixer: Exponential Expressive Gap For NLP Problems](https://mlsanity.com/paper/2208.08191/) (2022)
- [Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs](https://mlsanity.com/paper/2307.14988/) (2023)
- [Reduced Order Models and Conditional Expectation -- Analysing Parametric Low-Order Approximations](https://mlsanity.com/paper/2412.19836/) (2024)

## Abstract (arXiv)

The driving force behind convolutional networks - the most successful deep learning architecture to date, is their expressive power. Despite its wide acceptance and vast empirical evidence, formal analyses supporting this belief are scarce. The primary notions for formally reasoning about expressiveness are efficiency and inductive bias. Expressive efficiency refers to the ability of a network architecture to realize functions that require an alternative architecture to be much larger. Inductive bias refers to the prioritization of some functions over others given prior knowledge regarding a task at hand. In this paper we overview a series of works written by the authors, that through an equivalence to hierarchical tensor decompositions, analyze the expressive efficiency and inductive bias of various convolutional network architectural features (depth, width, strides and more). The results presented shed light on the demonstrated effectiveness of convolutional networks, and in addition, provide new tools for network design.