A scaling law of contextual persistence in human language
Not on the map. Why?
Abstract (arXiv)
Human language exhibits lawful structure at the level of words (frequency, vocabulary growth) and word pairs (co-occurrence across distance). Here we show that the arrangement of words in sequence -- a central determinant of meaning -- obeys a comparable law. Using large language models as probabilistic probes, we measured the reduction in target perplexity conferred by prior context at distance d beyond that of the same words scrambled; this difference, the contextual persistence function P(d), isolates the influence of arrangement. Across ten corpora spanning six language families and written and spoken modalities, P(d) decayed approximately as 1/d (P(d) propto d-α, mean α = 1.04; median r2 = 0.96). The effect vanished in scrambled and synthetic controls, replicated across independent probes, and did not appear in genomic or protein sequences under domain-native models. An exponent near 1 distributes contextual influence approximately uniformly across logarithmic timescales. The results establish a scaling law of contextual persistence in human language.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 42 references that are in the map. First posted on arXiv 28 Jul 2026.
Builds on
8 of 42 references.
References · 8
- 2017Attention Is All You Needcited by 60,361
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Mistral 7Bcited by 3,164
- 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
- 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
- 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 194
- 2016Criticality in Formal Languages and Statistical Physicscited by 35
- 2015Statistical laws in linguisticscited by 13
Built on by
Not cited yet.