# The magnitude of categories of texts enriched by language models

arXiv: https://arxiv.org/abs/2501.06662 · HTML: https://mlsanity.com/paper/2501.06662/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 35 references that are in the map.
- First posted on arXiv 11 Jan 2025.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An enriched category theory of language: from syntax to semantics](https://mlsanity.com/paper/2106.07890/) (2021)
- [Meaning Representations from Trajectories in Autoregressive Models](https://mlsanity.com/paper/2310.18348/) (2023)
- [Weighting vectors for machine learning: numerical harmonic analysis applied to boundary detection](https://mlsanity.com/paper/2106.00827/) (2021)
- [The magnitude vector of images](https://mlsanity.com/paper/2110.15188/) (2021)
- [Directed Metric Structures arising in Large Language Models](https://mlsanity.com/paper/2405.12264/) (2024)

## Built on by (most cited first, 5 of 5)

- [Token embeddings violate the manifold hypothesis](https://mlsanity.com/paper/2504.01002/) (2025)
- [Probing the topology of the space of tokens with structured prompts](https://mlsanity.com/paper/2503.15421/) (2025)
- [The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety](https://mlsanity.com/paper/2601.06116/) (2026)
- [TokenBlowUp: Resolving Representational Singularities in LLM Token Spaces via Monoidal Transformations](https://mlsanity.com/paper/2507.19747/) (2025)
- [Measuring What Persists: Conditioning Mechanisms and a Geometric Framework for AI Agent Identity](https://mlsanity.com/paper/2606.21843/) (2026)

## Abstract (arXiv)

The purpose of this article is twofold. Firstly, we use the next-token probabilities given by a language model to explicitly define a category of texts in natural language enriched over the unit interval, in the sense of Bradley, Terilla, and Vlassopoulos. We consider explicitly the terminating conditions for text generation and determine when the enrichment itself can be interpreted as a probability over texts. Secondly, we compute the M"obius function and the magnitude of an associated generalized metric space of texts. The magnitude function of that space is a sum over texts (prompts) of the t-logarithmic (Tsallis) entropies of the next-token probability distributions associated with each prompt, plus the cardinality of the model's possible outputs. A suitable evaluation of the magnitude function's derivative recovers a sum of Shannon entropies, which justifies seeing magnitude as a partition function. Following Leinster and Shulman, we also express the magnitude function of the generalized metric space as an Euler characteristic of magnitude homology and provide an explicit description of the zeroeth and first magnitude homology groups.