# Language Modeling with Reduced Densities

arXiv: https://arxiv.org/abs/2007.03834 · HTML: https://mlsanity.com/paper/2007.03834/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jul 2020.

## Built on by (most cited first, 5 of 5)

- [Grokking phase transitions in learning local rules with gradient descent](https://mlsanity.com/paper/2210.15435/) (2022)
- [An enriched category theory of language: from syntax to semantics](https://mlsanity.com/paper/2106.07890/) (2021)
- [Deep tensor networks with matrix product operators](https://mlsanity.com/paper/2209.09098/) (2022)
- [Quantum causal inference in the presence of hidden common causes: An entropic approach](https://mlsanity.com/paper/2104.13227/) (2021)
- [Density Matrices for Metaphor Understanding](https://mlsanity.com/paper/2408.11846/) (2024)

## Abstract (arXiv)

This work originates from the observation that today's state-of-the-art statistical language models are impressive not only for their performance, but also - and quite crucially - because they are built entirely from correlations in unstructured text data. The latter observation prompts a fundamental question that lies at the heart of this paper: What mathematical structure exists in unstructured text data? We put forth enriched category theory as a natural answer. We show that sequences of symbols from a finite alphabet, such as those found in a corpus of text, form a category enriched over probabilities. We then address a second fundamental question: How can this information be stored and modeled in a way that preserves the categorical structure? We answer this by constructing a functor from our enriched category of text to a particular enriched category of reduced density operators. The latter leverages the Loewner order on positive semidefinite operators, which can further be interpreted as a toy example of entailment.