mlsanityData to 14 Oct 2026

Entropy and type-token ratio in gigaword corpora

arXiv 2411.10227 ↗First posted 15 Nov 2024cs.CL cs.IR physics.soc-phcited by 6

Not on the map. Why?

Abstract (arXiv)

There are different ways of measuring diversity in complex systems. In particular, in language, lexical diversity is characterized in terms of the type-token ratio and the word entropy. We here investigate both diversity metrics in six massive linguistic datasets in English, Spanish, and Turkish, consisting of books, news articles, and tweets. These gigaword corpora correspond to languages with distinct morphological features and differ in registers and genres, thus constituting a varied testbed for a quantitative approach to lexical diversity. We unveil an empirical functional relation between entropy and type-token ratio of texts of a given corpus and language, which is a consequence of the statistical laws observed in natural language. Further, in the limit of large text lengths we find an analytical expression for this relation relying on both Zipf and Heaps laws that agrees with our empirical findings.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 19 of its 101 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

19 of 101 references.

References · 19

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  3. 2018A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguisticscited by 64
  4. 2020Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twittercited by 15
  5. 2015Statistical laws in linguisticscited by 13
  6. 2022Can Pre-trained Language Models Interpret Similes as Smart as Human?cited by 13
  7. 2022Estimating the Entropy of Linguistic Distributionscited by 12
  8. 2024Complex systems approach to natural languagecited by 12
  9. 2014Crowdsourcing Dialect Characterization through Twittercited by 11
  10. 2016In narrative texts punctuation marks obey the same statistics as wordscited by 11
  11. 2018Socioeconomic Dependencies of Linguistic Patterns in Twitter: A Multivariate Analysiscited by 9
  12. 2015Similarity of symbol frequency distributions with heavy tailscited by 6
7 more
  1. 2017Mapping the Americanization of English in Space and Timecited by 6
  2. 2021Regionalized models for Spanish language variations based on Twittercited by 4
  3. 2022Ordinal analysis of lexical patternscited by 4
  4. 2016Generalized Entropies and the Similarity of Textscited by 3
  5. 2021Capturing the diversity of multilingual societiescited by 3
  6. 2023When Dialects Collide: How Socioeconomic Mixing Affects Language Usecited by 2
  7. 2023Syntactic Variation Across the Grammar: Modelling a Complex Adaptive Systemcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Generative Exaggeration in LLM Social Agents: Consistency, Bias, and Toxicitycited by 9
  2. 2025RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Setscited by 1
  3. 2026LuxBorrow: From Pompier to Pompjee, Tracing Borrowing in Luxembourgishcited by 1
  4. 2026Generating High Quality Synthetic Data for Dutch Medical Conversationscited by 0
  5. 2026Detecting Historical Turning Points in Italian Media: A Complex Systems Approach to a Diachronic News Corpuscited by 0
  6. 2026Psychosis involves a deficit of information compression in connected speechcited by 0