# Tensor networks and efficient descriptions of classical data

arXiv: https://arxiv.org/abs/2103.06872 · HTML: https://mlsanity.com/paper/2103.06872/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 104 references that are in the map.
- First posted on arXiv 11 Mar 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications](https://mlsanity.com/paper/1701.05517/) (2017)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Why does deep and cheap learning work so well?](https://mlsanity.com/paper/1608.08225/) (2016)
- [On the Expressive Power of Deep Learning: A Tensor Analysis](https://mlsanity.com/paper/1509.05009/) (2015)
- [Neural Autoregressive Distribution Estimation](https://mlsanity.com/paper/1605.02226/) (2016)
- [Unsupervised Generative Modeling Using Matrix Product States](https://mlsanity.com/paper/1709.01662/) (2017)
- [Towards Quantum Machine Learning with Tensor Networks](https://mlsanity.com/paper/1803.11537/) (2018)
- [Tree Tensor Networks for Generative Modeling](https://mlsanity.com/paper/1901.02217/) (2019)
- [Deep AutoRegressive Networks](https://mlsanity.com/paper/1310.8499/) (2013)
- [An exact mapping between the Variational Renormalization Group and Deep Learning](https://mlsanity.com/paper/1410.3831/) (2014)
- [Expressive power of tensor-network factorizations for probabilistic modeling, with applications from hidden Markov models to quantum machine learning](https://mlsanity.com/paper/1907.03741/) (2019)
- [Learning Relevant Features of Data with Multi-scale Tensor Networks](https://mlsanity.com/paper/1801.00315/) (2017)
- [Quantum Entanglement in Deep Learning Architectures](https://mlsanity.com/paper/1803.09780/) (2018)
- [Criticality in Formal Languages and Statistical Physics](https://mlsanity.com/paper/1606.06737/) (2016)
- [Mutual Information, Neural Networks and the Renormalization Group](https://mlsanity.com/paper/1704.06279/) (2017)
- [Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines](https://mlsanity.com/paper/1712.04144/) (2017)
- [From probabilistic graphical models to generalized tensor networks for supervised learning](https://mlsanity.com/paper/1806.05964/) (2018)
- [Compressing deep neural networks by matrix product operators](https://mlsanity.com/paper/1904.06194/) (2019)
- [Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design](https://mlsanity.com/paper/1704.01552/) (2017)
- [Duality of Graphical Models and Tensor Networks](https://mlsanity.com/paper/1710.01437/) (2017)

## Built on by (most cited first, 13 of 13)

- [Tensor networks for quantum machine learning](https://mlsanity.com/paper/2303.11735/) (2023)
- [Generative modeling with projected entangled-pair states](https://mlsanity.com/paper/2202.08177/) (2022)
- [Generative Learning of Continuous Data by Tensor Networks](https://mlsanity.com/paper/2310.20498/) (2023)
- [L2M: Mutual Information Scaling Law for Long-Context Language Modeling](https://mlsanity.com/paper/2503.04725/) (2025)
- [Local Diffusion Models and Phases of Data Distributions](https://mlsanity.com/paper/2508.06614/) (2025)
- [Probabilistic Graphical Models and Tensor Networks: A Hybrid Framework](https://mlsanity.com/paper/2106.15666/) (2021)
- [Tensor network to learn the wavefunction of data](https://mlsanity.com/paper/2111.08014/) (2021)
- [What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglement](https://mlsanity.com/paper/2303.11249/) (2023)
- [Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks](https://mlsanity.com/paper/2505.20132/) (2025)
- [Generalization and Overfitting in Matrix Product State Machine Learning Architectures](https://mlsanity.com/paper/2208.04372/) (2022)
- [Differentiable Programming of Isometric Tensor Networks](https://mlsanity.com/paper/2110.03898/) (2021)
- [Deep tensor networks with matrix product operators](https://mlsanity.com/paper/2209.09098/) (2022)
- [Distributive Pre-Training of Generative Modeling Using Matrix-Product States](https://mlsanity.com/paper/2306.14787/) (2023)

## Abstract (arXiv)

We investigate the potential of tensor network based machine learning methods to scale to large image and text data sets. For that, we study how the mutual information between a subregion and its complement scales with the subsystem size L, similarly to how it is done in quantum many-body physics. We find that for text, the mutual information scales as a power law L^nu with a close to volume law exponent, indicating that text cannot be efficiently described by 1D tensor networks. For images, the scaling is close to an area law, hinting at 2D tensor networks such as PEPS could have an adequate expressibility. For the numerical analysis, we introduce a mutual information estimator based on autoregressive networks, and we also use convolutional neural networks in a neural estimator method.