# An enriched category theory of language: from syntax to semantics

arXiv: https://arxiv.org/abs/2106.07890 · HTML: https://mlsanity.com/paper/2106.07890/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 67 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Tropical Geometry of Deep Neural Networks](https://mlsanity.com/paper/1805.07091/) (2018)
- [A Tropical Approach to Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/1805.08749/) (2018)
- [A Survey on Neural Network Language Models](https://mlsanity.com/paper/1906.03591/) (2019)
- [Tropical Polynomial Division and Neural Networks](https://mlsanity.com/paper/1911.12922/) (2019)
- [Language Modeling with Reduced Densities](https://mlsanity.com/paper/2007.03834/) (2020)

## Built on by (most cited first, 22 of 22)

- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [On the Power of Foundation Models](https://mlsanity.com/paper/2211.16327/) (2022)
- [Neuro-Symbolic Causal Reasoning Meets Signaling Game for Emergent Semantic Communications](https://mlsanity.com/paper/2210.12040/) (2022)
- [Meaning Representations from Trajectories in Autoregressive Models](https://mlsanity.com/paper/2310.18348/) (2023)
- [Token embeddings violate the manifold hypothesis](https://mlsanity.com/paper/2504.01002/) (2025)
- [Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory](https://mlsanity.com/paper/2310.06756/) (2023)
- [The magnitude of categories of texts enriched by language models](https://mlsanity.com/paper/2501.06662/) (2025)
- [Unifying Causal Inference and Reinforcement Learning using Higher-Order Category Theory](https://mlsanity.com/paper/2209.06262/) (2022)
- [Quantum-Like Contextuality in Large Language Models](https://mlsanity.com/paper/2412.16806/) (2024)
- [Categoroids: Universal Conditional Independence](https://mlsanity.com/paper/2208.11077/) (2022)
- [Probing the topology of the space of tokens with structured prompts](https://mlsanity.com/paper/2503.15421/) (2025)
- [Enriching Disentanglement: From Logical Definitions to Quantitative Metrics](https://mlsanity.com/paper/2305.11512/) (2023)
- [Directed Metric Structures arising in Large Language Models](https://mlsanity.com/paper/2405.12264/) (2024)
- [Topos Theory for Generative AI and LLMs](https://mlsanity.com/paper/2508.08293/) (2025)
- [The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety](https://mlsanity.com/paper/2601.06116/) (2026)
- [A Rose by Any Other Name Would Smell as Sweet: Categorical Homotopy Theory for Large Language Models](https://mlsanity.com/paper/2508.10018/) (2025)
- [LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks](https://mlsanity.com/paper/2609.34187/) (2026)
- [Consciousness as a Functor](https://mlsanity.com/paper/2508.17561/) (2025)
- [Transparent Semantic Spaces: A Categorical Approach to Explainable Word Embeddings](https://mlsanity.com/paper/2508.20701/) (2025)
- [From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents](https://mlsanity.com/paper/2603.25342/) (2026)
- [Measuring What Persists: Conditioning Mechanisms and a Geometric Framework for AI Agent Identity](https://mlsanity.com/paper/2606.21843/) (2026)
- [When the Canonical Completion Is Wrong: Formalizing and Measuring the Jump in Large Language Models](https://mlsanity.com/paper/2608.26187/) (2026)

## Abstract (arXiv)

State of the art language models return a natural language text continuation from any piece of input text. This ability to generate coherent text extensions implies significant sophistication, including a knowledge of grammar and semantics. In this paper, we propose a mathematical framework for passing from probability distributions on extensions of given texts, such as the ones learned by today's large language models, to an enriched category containing semantic information. Roughly speaking, we model probability distributions on texts as a category enriched over the unit interval. Objects of this category are expressions in language, and hom objects are conditional probabilities that one expression is an extension of another. This category is syntactical -- it describes what goes with what. Then, via the Yoneda embedding, we pass to the enriched category of unit interval-valued copresheaves on this syntactical category. This category of enriched copresheaves is semantic -- it is where we find meaning, logical operations such as entailment, and the building blocks for more elaborate semantic concepts.