Two halves of a meaningful text are statistically different
Not on the map. Why?
Abstract (arXiv)
Which statistical features distinguish a meaningful text (possibly written in an unknown system) from a meaningless set of symbols? Here we answer this question by comparing features of the first half of a text to its second half. This comparison can uncover hidden effects, because the halves have the same values of many parameters (style, genre it etc). We found that the first half has more different words and more rare words than the second half. Also, words in the first half are distributed less homogeneously over the text in the sense of of the difference between the frequency and the inverse spatial period. These differences hold for the significant majority of several hundred relatively short texts we studied. The statistical significance is confirmed via the Wilcoxon test. Differences disappear after random permutation of words that destroys the linear structure of the text. The differences reveal a temporal asymmetry in meaningful texts, which is confirmed by showing that texts are much better compressible in their natural way (i.e. along the narrative) than in the word-inverted form. We conjecture that these results connect the semantic organization of a text (defined by the flow of its narrative) to its statistical features.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Apr 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 1.
4 papers.
- 2024Complex systems approach to natural languagecited by 12
- 2022Optimal alphabet for single text compressioncited by 1
- 2025Does a Large Language Model Really Speak in Human-Like Language?cited by 1
- 2023Unsupervised extraction of local and global keywords from a single textcited by 0
- 2025Does a Large Language Model Really Speak in Human-Like Language?2 Jan 2025
- 2024Complex systems approach to natural language5 Jan 2024
- 2023Unsupervised extraction of local and global keywords from a single text26 Jul 2023
- 2022Optimal alphabet for single text compression13 Jan 2022