# Entropy and type-token ratio in gigaword corpora

arXiv: https://arxiv.org/abs/2411.10227 · HTML: https://mlsanity.com/paper/2411.10227/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 101 references that are in the map.
- First posted on arXiv 15 Nov 2024.

## Builds on (most cited first, 19 of 19)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics](https://mlsanity.com/paper/1812.08092/) (2018)
- [Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twitter](https://mlsanity.com/paper/2007.12988/) (2020)
- [Statistical laws in linguistics](https://mlsanity.com/paper/1502.03296/) (2015)
- [Can Pre-trained Language Models Interpret Similes as Smart as Human?](https://mlsanity.com/paper/2203.08452/) (2022)
- [Estimating the Entropy of Linguistic Distributions](https://mlsanity.com/paper/2204.01469/) (2022)
- [Complex systems approach to natural language](https://mlsanity.com/paper/2401.02772/) (2024)
- [Crowdsourcing Dialect Characterization through Twitter](https://mlsanity.com/paper/1407.7094/) (2014)
- [In narrative texts punctuation marks obey the same statistics as words](https://mlsanity.com/paper/1604.00834/) (2016)
- [Socioeconomic Dependencies of Linguistic Patterns in Twitter: A Multivariate Analysis](https://mlsanity.com/paper/1804.01155/) (2018)
- [Similarity of symbol frequency distributions with heavy tails](https://mlsanity.com/paper/1510.00277/) (2015)
- [Mapping the Americanization of English in Space and Time](https://mlsanity.com/paper/1707.00781/) (2017)
- [Regionalized models for Spanish language variations based on Twitter](https://mlsanity.com/paper/2110.06128/) (2021)
- [Ordinal analysis of lexical patterns](https://mlsanity.com/paper/2208.11175/) (2022)
- [Generalized Entropies and the Similarity of Texts](https://mlsanity.com/paper/1611.03596/) (2016)
- [Capturing the diversity of multilingual societies](https://mlsanity.com/paper/2105.02570/) (2021)
- [When Dialects Collide: How Socioeconomic Mixing Affects Language Use](https://mlsanity.com/paper/2307.10016/) (2023)
- [Syntactic Variation Across the Grammar: Modelling a Complex Adaptive System](https://mlsanity.com/paper/2309.11869/) (2023)

## Built on by (most cited first, 6 of 6)

- [Generative Exaggeration in LLM Social Agents: Consistency, Bias, and Toxicity](https://mlsanity.com/paper/2507.00657/) (2025)
- [RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Sets](https://mlsanity.com/paper/2510.06143/) (2025)
- [LuxBorrow: From Pompier to Pompjee, Tracing Borrowing in Luxembourgish](https://mlsanity.com/paper/2603.10789/) (2026)
- [Generating High Quality Synthetic Data for Dutch Medical Conversations](https://mlsanity.com/paper/2604.09645/) (2026)
- [Detecting Historical Turning Points in Italian Media: A Complex Systems Approach to a Diachronic News Corpus](https://mlsanity.com/paper/2606.14348/) (2026)
- [Psychosis involves a deficit of information compression in connected speech](https://mlsanity.com/paper/2609.15522/) (2026)

## Abstract (arXiv)

There are different ways of measuring diversity in complex systems. In particular, in language, lexical diversity is characterized in terms of the type-token ratio and the word entropy. We here investigate both diversity metrics in six massive linguistic datasets in English, Spanish, and Turkish, consisting of books, news articles, and tweets. These gigaword corpora correspond to languages with distinct morphological features and differ in registers and genres, thus constituting a varied testbed for a quantitative approach to lexical diversity. We unveil an empirical functional relation between entropy and type-token ratio of texts of a given corpus and language, which is a consequence of the statistical laws observed in natural language. Further, in the limit of large text lengths we find an analytical expression for this relation relying on both Zipf and Heaps laws that agrees with our empirical findings.