mlsanityData to 14 Oct 2026

Tensor networks and efficient descriptions of classical data

arXiv 2103.06872 ↗First posted 11 Mar 2021quant-ph cond-mat.str-el cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

We investigate the potential of tensor network based machine learning methods to scale to large image and text data sets. For that, we study how the mutual information between a subregion and its complement scales with the subsystem size L, similarly to how it is done in quantum many-body physics. We find that for text, the mutual information scales as a power law L^nu with a close to volume law exponent, indicating that text cannot be efficiently described by 1D tensor networks. For images, the scaling is close to an area law, hinting at 2D tensor networks such as PEPS could have an adequate expressibility. For the numerical analysis, we introduce a mutual information estimator based on autoregressive networks, and we also use convolutional neural networks in a neural estimator method.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 33 of its 104 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

33 of 104 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  4. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  5. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  6. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  7. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  8. 2018MINE: Mutual Information Neural Estimationcited by 683
  9. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 417
  10. 2019On Mutual Information Maximization for Representation Learningcited by 316
  11. 2015Tensorizing Neural Networkscited by 250
  12. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
21 more
  1. 2016Why does deep and cheap learning work so well?cited by 100
  2. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  3. 2016Neural Autoregressive Distribution Estimationcited by 95
  4. 2017Unsupervised Generative Modeling Using Matrix Product Statescited by 88
  5. 2018Towards Quantum Machine Learning with Tensor Networkscited by 76
  6. 2019Tree Tensor Networks for Generative Modelingcited by 64
  7. 2013Deep AutoRegressive Networkscited by 61
  8. 2014An exact mapping between the Variational Renormalization Group and Deep Learningcited by 56
  9. 2019Expressive power of tensor-network factorizations for probabilistic modeling, with applications from hidden Markov models to quantum machine learningcited by 55
  10. 2017Learning Relevant Features of Data with Multi-scale Tensor Networkscited by 51
  11. 2018Quantum Entanglement in Deep Learning Architecturescited by 50
  12. 2016Criticality in Formal Languages and Statistical Physicscited by 35
  13. 2017Mutual Information, Neural Networks and the Renormalization Groupcited by 33
  14. 2017Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machinescited by 33
  15. 2018From probabilistic graphical models to generalized tensor networks for supervised learningcited by 32
  16. 2019Compressing deep neural networks by matrix product operatorscited by 28
  17. 2017Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Designcited by 24
  18. 2017Duality of Graphical Models and Tensor Networkscited by 15
  19. 2019Mutual Information Scaling and Expressive Power of Sequence Modelscited by 14
  20. 2017Language Design as Information Renormalizationcited by 6
  21. 2017Entanglement Entropy of Target Functions for Image Classification and Convolutional Neural Networkcited by 5

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 3; 2023: 4; 2024: 0; 2025: 3.

13 papers.

Sort:

By area:

  1. 2023Tensor networks for quantum machine learningcited by 17
  2. 2022Generative modeling with projected entangled-pair statescited by 16
  3. 2023Generative Learning of Continuous Data by Tensor Networkscited by 9
  4. 2025L2M: Mutual Information Scaling Law for Long-Context Language Modelingcited by 9
  5. 2025Local Diffusion Models and Phases of Data Distributionscited by 8
  6. 2021Probabilistic Graphical Models and Tensor Networks: A Hybrid Frameworkcited by 5
  7. 2021Tensor network to learn the wavefunction of datacited by 4
  8. 2023What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglementcited by 4
  9. 2025Tensorization is a powerful but underexplored tool for compression and interpretability of neural networkscited by 4
  10. 2022Generalization and Overfitting in Matrix Product State Machine Learning Architecturescited by 3
  11. 2021Differentiable Programming of Isometric Tensor Networkscited by 2
  12. 2022Deep tensor networks with matrix product operatorscited by 2
1 more
  1. 2023Distributive Pre-Training of Generative Modeling Using Matrix-Product Statescited by 2