# Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse

arXiv: https://arxiv.org/abs/2410.16062 · HTML: https://mlsanity.com/paper/2410.16062/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 89 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [The emotional arcs of stories are dominated by six basic shapes](https://mlsanity.com/paper/1606.07772/) (2016)
- [Revisiting the Uniform Information Density Hypothesis](https://mlsanity.com/paper/2109.11635/) (2021)
- [Generalized Measures of Anticipation and Responsivity in Online Language Processing](https://mlsanity.com/paper/2409.10728/) (2024)
- [Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives](https://mlsanity.com/paper/2310.13676/) (2023)
- [Cross-lingual RST Discourse Parsing](https://mlsanity.com/paper/1701.02946/) (2017)
- [On the Role of Context in Reading Time Prediction](https://mlsanity.com/paper/2409.08160/) (2024)
- [A Cross-Linguistic Pressure for Uniform Information Density in Word Order](https://mlsanity.com/paper/2306.03734/) (2023)
- [Disambiguatory Signals are Stronger in Word-initial Positions](https://mlsanity.com/paper/2102.02183/) (2021)
- [Revisiting Entropy Rate Constancy in Text](https://mlsanity.com/paper/2305.12084/) (2023)
- [From "um" to "yeah": Producing, predicting, and regulating information flow in human conversation](https://mlsanity.com/paper/2403.08890/) (2024)
- [Controlling Surprisal in Music Generation via Information Content Curve Matching](https://mlsanity.com/paper/2408.06022/) (2024)
- [Construction Repetition Reduces Information Rate in Dialogue](https://mlsanity.com/paper/2210.08321/) (2022)
- [How do decoding algorithms distribute information in dialogue responses?](https://mlsanity.com/paper/2303.17006/) (2023)
- [An Exploration of Left-Corner Transformations](https://mlsanity.com/paper/2311.16258/) (2023)

## Built on by (most cited first, 17 of 17)

- [Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMs](https://mlsanity.com/paper/2502.18795/) (2025)
- [The Harmonic Structure of Information Contours](https://mlsanity.com/paper/2506.03902/) (2025)
- [Identifying the Periodicity of Information in Natural Language](https://mlsanity.com/paper/2510.27241/) (2025)
- [Is Information Density Uniform when Utterances are Grounded on Perception and Discourse?](https://mlsanity.com/paper/2602.14653/) (2026)
- [Measuring and Mitigating Post-hoc Rationalization in Reverse Chain-of-Thought Generation](https://mlsanity.com/paper/2602.14469/) (2026)
- [On the Proper Treatment of Units in Surprisal Theory](https://mlsanity.com/paper/2604.28147/) (2026)
- [From Input Perception to Predictive Insight: Modeling Model Blind Spots Before They Become Errors](https://mlsanity.com/paper/2509.20065/) (2025)
- [Predicting States of Understanding in Explanatory Interactions Using Cognitive Load-Related Linguistic Cues](https://mlsanity.com/paper/2603.20079/) (2026)
- [Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialogue](https://mlsanity.com/paper/2605.00506/) (2026)
- [Revisiting the UID Hypothesis in LLM Reasoning Traces](https://mlsanity.com/paper/2510.13850/) (2025)
- [Expect the Unexpected? Testing the Surprisal of Salient Entities](https://mlsanity.com/paper/2604.10724/) (2026)
- [CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics](https://mlsanity.com/paper/2604.15647/) (2026)
- [The Dynamics of Human and AI-Generated Language: How Semantics Fluctuates across Different Timescales](https://mlsanity.com/paper/2606.11371/) (2026)
- [Using LMs to Model the Effects of Context and Coreference during Sentence Comprehension](https://mlsanity.com/paper/2609.32119/) (2026)
- [Uniform Information Density-Based Preference Optimization](https://mlsanity.com/paper/2610.14578/) (2026)
- [An Interpretable Linguistically-Grounded Analysis of Scene Saliency in Movie Screenplays](https://mlsanity.com/paper/2610.15544/) (2026)
- [Worth Their Weights? Evaluating Generative Models for Discourse Parsing](https://mlsanity.com/paper/2610.17658/) (2026)

## Abstract (arXiv)

The Uniform Information Density (UID) hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. Of course, information rate in texts and discourses is not perfectly uniform. While these fluctuations can be viewed as theoretically uninteresting noise on top of a uniform target, another explanation is that UID is not the only functional pressure regulating information content in a language. Speakers may also seek to maintain interest, adhere to writing conventions, and build compelling arguments. In this paper, we propose one such functional pressure; namely that speakers modulate information rate based on location within a hierarchically-structured model of discourse. We term this the Structured Context Hypothesis and test it by predicting the surprisal contours of naturally occurring discourses extracted from large language models using predictors derived from discourse structure. We find that hierarchical predictors are significant predictors of a discourse's information contour and that deeply nested hierarchical predictors are more predictive than shallow ones. This work takes an initial step beyond UID to propose testable hypotheses for why the information rate fluctuates in predictable ways