# Revisiting the Uniform Information Density Hypothesis arXiv: https://arxiv.org/abs/2109.11635 ยท HTML: https://mlsanity.com/paper/2109.11635/ ## Key facts - Cited by 68 papers in the map (data through 14 Oct 2026). - Builds on 12 of its 60 references that are in the map. - First posted on arXiv 23 Sep 2021. ## Builds on (most cited first, 12 of 12) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019) - [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018) - [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020) - [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020) - [As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages](https://mlsanity.com/paper/2012.05628/) (2020) - [Lower Perplexity is Not Always Human-Like](https://mlsanity.com/paper/2106.01229/) (2021) - [The Natural Stories Corpus](https://mlsanity.com/paper/1708.05763/) (2017) - [A Cognitive Regularizer for Language Modeling](https://mlsanity.com/paper/2105.07144/) (2021) - [Availability-Based Production Predicts Speakers' Real-time Choices of Mandarin Classifiers](https://mlsanity.com/paper/1905.07321/) (2019) ## Built on by (most cited first, 30 of 68) - [Truncation Sampling as Language Model Desmoothing](https://mlsanity.com/paper/2210.15191/) (2022) - [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023) - [Efficient Transformers with Dynamic Token Pooling](https://mlsanity.com/paper/2211.09761/) (2022) - [GPT-who: An Information Density-based Machine-Generated Text Detector](https://mlsanity.com/paper/2310.06202/) (2023) - [A Measure-Theoretic Characterization of Tight Language Models](https://mlsanity.com/paper/2212.10502/) (2022) - [LLM Probability Concentration: How Alignment Shrinks the Generative Horizon](https://mlsanity.com/paper/2506.17871/) (2025) - [A surprisal--duration trade-off across and within the world's languages](https://mlsanity.com/paper/2109.15000/) (2021) - [Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives](https://mlsanity.com/paper/2310.13676/) (2023) - [On the Role of Context in Reading Time Prediction](https://mlsanity.com/paper/2409.08160/) (2024) - [Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs](https://mlsanity.com/paper/2502.18795/) (2025) - [On the Effect of Anticipation on Reading Times](https://mlsanity.com/paper/2211.14301/) (2022) - [A Cross-Linguistic Pressure for Uniform Information Density in Word Order](https://mlsanity.com/paper/2306.03734/) (2023) - [Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse](https://mlsanity.com/paper/2410.16062/) (2024) - [On the Proper Treatment of Tokenization in Psycholinguistics](https://mlsanity.com/paper/2410.02691/) (2024) - [Estimating the Entropy of Linguistic Distributions](https://mlsanity.com/paper/2204.01469/) (2022) - [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026) - [Analyzing Wrap-Up Effects through an Information-Theoretic Lens](https://mlsanity.com/paper/2203.17213/) (2022) - [What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length](https://mlsanity.com/paper/2411.02528/) (2024) - [Language Models Grow Less Humanlike beyond Phase Transition](https://mlsanity.com/paper/2502.18802/) (2025) - [Towards a Similarity-adjusted Surprisal Theory](https://mlsanity.com/paper/2410.17676/) (2024) - [The Harmonic Structure of Information Contours](https://mlsanity.com/paper/2506.03902/) (2025) - [Revisiting Entropy Rate Constancy in Text](https://mlsanity.com/paper/2305.12084/) (2023) - [Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics](https://mlsanity.com/paper/2404.19178/) (2024) - [Measuring AI "Slop" in Text](https://mlsanity.com/paper/2509.19163/) (2025) - [Revisiting the Optimality of Word Lengths](https://mlsanity.com/paper/2312.03897/) (2023) - [The time scale of redundancy between prosody and linguistic context](https://mlsanity.com/paper/2503.11630/) (2025) - [Identifying the Periodicity of Information in Natural Language](https://mlsanity.com/paper/2510.27241/) (2025) - [How do decoding algorithms distribute information in dialogue responses?](https://mlsanity.com/paper/2303.17006/) (2023) - [Quantifying the perceptual value of lexical and non-lexical channels in speech](https://mlsanity.com/paper/2307.03534/) (2023) - [Is Information Density Uniform when Utterances are Grounded on Perception and Discourse?](https://mlsanity.com/paper/2602.14653/) (2026) ## Abstract (arXiv) The uniform information density (UID) hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal. While its implications on language production have been well explored, the hypothesis potentially makes predictions about language comprehension and linguistic acceptability as well. Further, it is unclear how uniformity in a linguistic signal -- or lack thereof -- should be measured, and over which linguistic unit, e.g., the sentence or language level, this uniformity should hold. Here we investigate these facets of the UID hypothesis using reading time and acceptability data. While our reading time results are generally consistent with previous work, they are also consistent with a weakly super-linear effect of surprisal, which would be compatible with UID's predictions. For acceptability judgments, we find clearer evidence that non-uniformity in information density is predictive of lower acceptability. We then explore multiple operationalizations of UID, motivated by different interpretations of the original hypothesis, and analyze the scope over which the pressure towards uniformity is exerted. The explanatory power of a subset of the proposed operationalizations suggests that the strongest trend may be a regression towards a mean surprisal across the language, rather than the phrase, sentence, or document -- a finding that supports a typical interpretation of UID, namely that it is the byproduct of language users maximizing the use of a (hypothetical) communication channel.