# Correlation Dimension of Natural Language in a Statistical Manifold

arXiv: https://arxiv.org/abs/2405.06321 · HTML: https://mlsanity.com/paper/2405.06321/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 10 May 2024.

## Builds on (most cited first, 7 of 7)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [Taylor's law for Human Linguistic Sequences](https://mlsanity.com/paper/1804.07893/) (2018)
- [The Fractional Preferential Attachment Scale-Free Network Model](https://mlsanity.com/paper/2005.01512/) (2020)
- [Elsevier OA CC-By Corpus](https://mlsanity.com/paper/2008.00774/) (2020)

## Built on by (most cited first, 2 of 2)

- [Correlation Dimension of Auto-Regressive Large Language Models](https://mlsanity.com/paper/2510.21258/) (2025)
- [Escaping Mode Collapse in LLM Generation via Geometric Regulation](https://mlsanity.com/paper/2605.00435/) (2026)

## Abstract (arXiv)

The correlation dimension of natural language is measured by applying the Grassberger-Procaccia algorithm to high-dimensional sequences produced by a large-scale language model. This method, previously studied only in a Euclidean space, is reformulated in a statistical manifold via the Fisher-Rao distance. Language exhibits a multifractal, with global self-similarity and a universal dimension around 6.5, which is smaller than those of simple discrete random sequences and larger than that of a Barab'asi-Albert process. Long memory is the key to producing self-similarity. Our method is applicable to any probabilistic model of real-world discrete sequences, and we show an application to music data.