# Explainable Natural Language Processing with Matrix Product States

arXiv: https://arxiv.org/abs/2112.08628 · HTML: https://mlsanity.com/paper/2112.08628/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 19 of 19)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Entailment as Few-Shot Learner](https://mlsanity.com/paper/2104.14690/) (2021)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [Expressive power of tensor-network factorizations for probabilistic modeling, with applications from hidden Markov models to quantum machine learning](https://mlsanity.com/paper/1907.03741/) (2019)
- [Learning Relevant Features of Data with Multi-scale Tensor Networks](https://mlsanity.com/paper/1801.00315/) (2017)
- [Quantum Entanglement in Deep Learning Architectures](https://mlsanity.com/paper/1803.09780/) (2018)
- [Machine Learning by Unitary Tensor Network of Hierarchical Tree Structure](https://mlsanity.com/paper/1710.04833/) (2017)
- [Criticality in Formal Languages and Statistical Physics](https://mlsanity.com/paper/1606.06737/) (2016)
- [From probabilistic graphical models to generalized tensor networks for supervised learning](https://mlsanity.com/paper/1806.05964/) (2018)
- [Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Design](https://mlsanity.com/paper/1704.01552/) (2017)
- [Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function](https://mlsanity.com/paper/2009.04007/) (2020)
- [Modeling Sequences with Quantum States: A Look Under the Hood](https://mlsanity.com/paper/1910.07425/) (2019)
- [On Multiplicative Integration with Recurrent Neural Networks](https://mlsanity.com/paper/1606.06630/) (2016)
- [On the Long-Term Memory of Deep Recurrent Networks](https://mlsanity.com/paper/1710.09431/) (2017)
- [Probabilistic Modeling with Matrix Product States](https://mlsanity.com/paper/1902.06888/) (2019)
- [Graph Star Net for Generalized Multi-Task Learning](https://mlsanity.com/paper/1906.12330/) (2019)

## Built on by (most cited first, 7 of 7)

- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [TensorKrowch: Smooth integration of tensor networks in machine learning](https://mlsanity.com/paper/2306.08595/) (2023)
- [Tensorization of neural networks for improved privacy and interpretability](https://mlsanity.com/paper/2501.06300/) (2025)
- [Enabling Quantum Natural Language Processing for Hindi Language](https://mlsanity.com/paper/2312.01221/) (2023)
- [Group-invariant tensor train networks for supervised learning](https://mlsanity.com/paper/2206.15051/) (2022)
- [Private and interpretable clinical prediction with quantum-inspired tensor train models](https://mlsanity.com/paper/2602.06110/) (2026)
- [Quantum-inspired tensor networks in machine learning models](https://mlsanity.com/paper/2604.14287/) (2026)

## Abstract (arXiv)

Despite empirical successes of recurrent neural networks (RNNs) in natural language processing (NLP), theoretical understanding of RNNs is still limited due to intrinsically complex non-linear computations. We systematically analyze RNNs' behaviors in a ubiquitous NLP task, the sentiment analysis of movie reviews, via the mapping between a class of RNNs called recurrent arithmetic circuits (RACs) and a matrix product state (MPS). Using the von-Neumann entanglement entropy (EE) as a proxy for information propagation, we show that single-layer RACs possess a maximum information propagation capacity, reflected by the saturation of the EE. Enlarging the bond dimension beyond the EE saturation threshold does not increase model prediction accuracies, so a minimal model that best estimates the data statistics can be inferred. Although the saturated EE is smaller than the maximum EE allowed by the area law, our minimal model still achieves ~99% training accuracies in realistic sentiment analysis data sets. Thus, low EE is not a warrant against the adoption of single-layer RACs for NLP. Contrary to a common belief that long-range information propagation is the main source of RNNs' successes, we show that single-layer RACs harness high expressiveness from the subtle interplay between the information propagation and the word vector embeddings. Our work sheds light on the phenomenology of learning in RACs, and more generally on the explainability of RNNs for NLP, using tools from many-body quantum physics.