# Revisiting the Optimality of Word Lengths

arXiv: https://arxiv.org/abs/2312.03897 · HTML: https://mlsanity.com/paper/2312.03897/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 44 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Revisiting the Uniform Information Density Hypothesis](https://mlsanity.com/paper/2109.11635/) (2021)
- [A surprisal--duration trade-off across and within the world's languages](https://mlsanity.com/paper/2109.15000/) (2021)
- [How (Non-)Optimal is the Lexicon?](https://mlsanity.com/paper/2104.14279/) (2021)
- [Direct and indirect evidence of compression of word lengths. Zipf's law of abbreviation revisited](https://mlsanity.com/paper/2303.10128/) (2023)

## Built on by (most cited first, 5 of 5)

- [How to Compute the Probability of a Word](https://mlsanity.com/paper/2406.14561/) (2024)
- [Using Information Theory to Characterize Prosodic Typology: The Case of Tone, Pitch-Accent and Stress-Accent](https://mlsanity.com/paper/2505.07659/) (2025)
- [Strategic resource allocation in memory encoding: An efficiency principle shaping language processing](https://mlsanity.com/paper/2503.14728/) (2025)
- [Is Structure Dependence Shaped for Efficient Communication?: A Case Study on Coordination](https://mlsanity.com/paper/2410.10556/) (2024)
- [SAMAS: A Spectrum-Guided Multi-Agent System for Achieving Style Fidelity in Literary Translation](https://mlsanity.com/paper/2602.19840/) (2026)

## Abstract (arXiv)

Zipf (1935) posited that wordforms are optimized to minimize utterances' communicative costs. Under the assumption that cost is given by an utterance's length, he supported this claim by showing that words' lengths are inversely correlated with their frequencies. Communicative cost, however, can be operationalized in different ways. Piantadosi et al. (2011) claim that cost should be measured as the distance between an utterance's information rate and channel capacity, which we dub the channel capacity hypothesis (CCH) here. Following this logic, they then proposed that a word's length should be proportional to the expected value of its surprisal (negative log-probability in context). In this work, we show that Piantadosi et al.'s derivation does not minimize CCH's cost, but rather a lower bound, which we term CCH-lower. We propose a novel derivation, suggesting an improved way to minimize CCH's cost. Under this method, we find that a language's word lengths should instead be proportional to the surprisal's expectation plus its variance-to-mean ratio. Experimentally, we compare these three communicative cost functions: Zipf's, CCH-lower , and CCH. Across 13 languages and several experimental settings, we find that length is better predicted by frequency than either of the other hypotheses. In fact, when surprisal's expectation, or expectation plus variance-to-mean ratio, is estimated using better language models, it leads to worse word length predictions. We take these results as evidence that Zipf's longstanding hypothesis holds.