# Autocorrelations Decay in Texts and Applicability Limits of Language Models

arXiv: https://arxiv.org/abs/2305.06615 · HTML: https://mlsanity.com/paper/2305.06615/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 May 2023.

## Built on by (most cited first, 9 of 9)

- [Towards a theory of how the structure of language is acquired by deep neural networks](https://mlsanity.com/paper/2406.00048/) (2024)
- [Phase transition in large language models and the criticality of natural languages](https://mlsanity.com/paper/2406.05335/) (2024)
- [States of LLM-generated Texts and Phase Transitions between them](https://mlsanity.com/paper/2503.06330/) (2025)
- [Rethinking the Relationship between the Power Law and Hierarchical Structures](https://mlsanity.com/paper/2505.04984/) (2025)
- [The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora](https://mlsanity.com/paper/2608.09093/) (2026)
- [Long Text Generation Challenge](https://mlsanity.com/paper/2306.02334/) (2023)
- [Vocabulary Growth Fundamentals: Bernstein Functions and Hausdorff Sequences](https://mlsanity.com/paper/2608.29449/) (2026)
- [Doc2FRC: Length-Consistent Document-Level Machine Translation via Fixed-Range Chunking](https://mlsanity.com/paper/2609.12674/) (2026)
- [Writing as a Self-Organized Critical Process](https://mlsanity.com/paper/2610.05466/) (2026)

## Abstract (arXiv)

We show that the laws of autocorrelations decay in texts are closely related to applicability limits of language models. Using distributional semantics we empirically demonstrate that autocorrelations of words in texts decay according to a power law. We show that distributional semantics provides coherent autocorrelations decay exponents for texts translated to multiple languages. The autocorrelations decay in generated texts is quantitatively and often qualitatively different from the literary texts. We conclude that language models exhibiting Markov behavior, including large autoregressive language models, may have limitations when applied to long texts, whether analysis or generation.