# Estimating the Entropy of Linguistic Distributions

arXiv: https://arxiv.org/abs/2204.01469 · HTML: https://mlsanity.com/paper/2204.01469/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 54 references that are in the map.
- First posted on arXiv 4 Apr 2022.

## Builds on (most cited first, 7 of 7)

- [Revisiting the Uniform Information Density Hypothesis](https://mlsanity.com/paper/2109.11635/) (2021)
- [Are All Languages Equally Hard to Language-Model?](https://mlsanity.com/paper/1806.03743/) (2018)
- [On the Complexity and Typology of Inflectional Morphological Systems](https://mlsanity.com/paper/1807.02747/) (2018)
- [Morphological Irregularity Correlates with Frequency](https://mlsanity.com/paper/1906.11483/) (2019)
- [How (Non-)Optimal is the Lexicon?](https://mlsanity.com/paper/2104.14279/) (2021)
- [On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs](https://mlsanity.com/paper/2005.01204/) (2020)
- [Modeling the Unigram Distribution](https://mlsanity.com/paper/2106.02289/) (2021)

## Built on by (most cited first, 12 of 12)

- [Selectively Answering Ambiguous Questions](https://mlsanity.com/paper/2305.14613/) (2023)
- [gzip Predicts Data-dependent Scaling Laws](https://mlsanity.com/paper/2405.16684/) (2024)
- [Entropy and type-token ratio in gigaword corpora](https://mlsanity.com/paper/2411.10227/) (2024)
- [PTVD: A Large-Scale Plot-Oriented Multimodal Dataset Based on Television Dramas](https://mlsanity.com/paper/2306.14644/) (2023)
- [The Causal Influence of Grammatical Gender on Distributional Semantics](https://mlsanity.com/paper/2311.18567/) (2023)
- [Rethinking the Relationship between the Power Law and Hierarchical Structures](https://mlsanity.com/paper/2505.04984/) (2025)
- [MASALA: Modelling and Analysing the Semantics of Adpositions in Linguistic Annotation of Hindi](https://mlsanity.com/paper/2205.03955/) (2022)
- [Surprisal reveals diversity gaps in image captioning and different scorers change the story](https://mlsanity.com/paper/2511.04754/) (2025)
- [Partitioning the Sample Space for a More Precise Shannon Entropy Estimation](https://mlsanity.com/paper/2512.10133/) (2025)
- [On the scaling relationship between cloze probabilities and language model next-token prediction](https://mlsanity.com/paper/2602.17848/) (2026)
- [Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs](https://mlsanity.com/paper/2609.16454/) (2026)
- [Revisiting the Estimation of Unigram Surprisal in Computational Psycholinguistics](https://mlsanity.com/paper/2610.18414/) (2026)

## Abstract (arXiv)

Shannon entropy is often a quantity of interest to linguists studying the communicative capacity of human language. However, entropy must typically be estimated from observed data because researchers do not have access to the underlying probability distribution that gives rise to these data. While entropy estimation is a well-studied problem in other fields, there is not yet a comprehensive exploration of the efficacy of entropy estimators for use with linguistic data. In this work, we fill this void, studying the empirical effectiveness of different entropy estimators for linguistic distributions. In a replication of two recent information-theoretic linguistic studies, we find evidence that the reported effect size is over-estimated due to over-reliance on poor entropy estimators. Finally, we end our paper with concrete recommendations for entropy estimation depending on distribution type and data availability.