# How (Non-)Optimal is the Lexicon?

arXiv: https://arxiv.org/abs/2104.14279 · HTML: https://mlsanity.com/paper/2104.14279/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 67 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 10 of 10)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [Consistency of a Recurrent Language Model With Respect to Incomplete Decoding](https://mlsanity.com/paper/2002.02492/) (2020)
- [Phonotactic Complexity and its Trade-offs](https://mlsanity.com/paper/2005.03774/) (2020)
- [Meaning to Form: Measuring Systematicity as Information](https://mlsanity.com/paper/1906.05906/) (2019)
- [Optimal coding and the origins of Zipfian laws](https://mlsanity.com/paper/1906.01545/) (2019)
- [Disambiguatory Signals are Stronger in Word-initial Positions](https://mlsanity.com/paper/2102.02183/) (2021)
- [Polysemy and brevity versus frequency in language](https://mlsanity.com/paper/1904.00812/) (2019)
- [The origins of Zipf's meaning-frequency law](https://mlsanity.com/paper/1801.00168/) (2017)
- [Finding Concept-specific Biases in Form--Meaning Associations](https://mlsanity.com/paper/2104.06325/) (2021)

## Built on by (most cited first, 11 of 11)

- [A Cross-Linguistic Pressure for Uniform Information Density in Word Order](https://mlsanity.com/paper/2306.03734/) (2023)
- [On the probability-quality paradox in language generation](https://mlsanity.com/paper/2203.17217/) (2022)
- [Estimating the Entropy of Linguistic Distributions](https://mlsanity.com/paper/2204.01469/) (2022)
- [Modeling the Unigram Distribution](https://mlsanity.com/paper/2106.02289/) (2021)
- [Revisiting the Optimality of Word Lengths](https://mlsanity.com/paper/2312.03897/) (2023)
- [Direct and indirect evidence of compression of word lengths. Zipf's law of abbreviation revisited](https://mlsanity.com/paper/2303.10128/) (2023)
- [Pragmatic Constraint on Distributional Semantics](https://mlsanity.com/paper/2211.11041/) (2022)
- [Word reuse and combination support efficient communication of emerging concepts](https://mlsanity.com/paper/2411.05379/) (2024)
- [The optimality of word lengths. Theoretical foundations and an empirical study](https://mlsanity.com/paper/2208.10384/) (2022)
- [Universal Topological Regularity of Syntactic Structures](https://mlsanity.com/paper/2302.00129/) (2023)
- [Dependency distance minimization predicts compression](https://mlsanity.com/paper/2109.08900/) (2021)

## Abstract (arXiv)

The mapping of lexical meanings to wordforms is a major feature of natural languages. While usage pressures might assign short words to frequent meanings (Zipf's law of abbreviation), the need for a productive and open-ended vocabulary, local constraints on sequences of symbols, and various other factors all shape the lexicons of the world's languages. Despite their importance in shaping lexical structure, the relative contributions of these factors have not been fully quantified. Taking a coding-theoretic view of the lexicon and making use of a novel generative statistical model, we define upper bounds for the compressibility of the lexicon under various constraints. Examining corpora from 7 typologically diverse languages, we use those upper bounds to quantify the lexicon's optimality and to explore the relative costs of major constraints on natural codes. We find that (compositional) morphology and graphotactics can sufficiently account for most of the complexity of natural codes -- as measured by code length.