# Towards a Similarity-adjusted Surprisal Theory

arXiv: https://arxiv.org/abs/2410.17676 · HTML: https://mlsanity.com/paper/2410.17676/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 44 references that are in the map.
- First posted on arXiv 23 Oct 2024.

## Builds on (most cited first, 12 of 12)

- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [Revisiting the Uniform Information Density Hypothesis](https://mlsanity.com/paper/2109.11635/) (2021)
- [The Natural Stories Corpus](https://mlsanity.com/paper/1708.05763/) (2017)
- [How to Compute the Probability of a Word](https://mlsanity.com/paper/2406.14561/) (2024)
- [Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities](https://mlsanity.com/paper/2210.12187/) (2022)
- [Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities](https://mlsanity.com/paper/2406.10851/) (2024)
- [Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words](https://mlsanity.com/paper/2005.01810/) (2020)
- [Generalized Measures of Anticipation and Responsivity in Online Language Processing](https://mlsanity.com/paper/2409.10728/) (2024)
- [Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives](https://mlsanity.com/paper/2310.13676/) (2023)
- [Modeling the Unigram Distribution](https://mlsanity.com/paper/2106.02289/) (2021)

## Built on by (most cited first, 9 of 9)

- [Large Language Models Are Human-Like Internally](https://mlsanity.com/paper/2502.01615/) (2025)
- [The Harmonic Structure of Information Contours](https://mlsanity.com/paper/2506.03902/) (2025)
- [Is Information Density Uniform when Utterances are Grounded on Perception and Discourse?](https://mlsanity.com/paper/2602.14653/) (2026)
- [Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobs](https://mlsanity.com/paper/2505.02072/) (2025)
- [Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal](https://mlsanity.com/paper/2601.09886/) (2026)
- [Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialogue](https://mlsanity.com/paper/2605.00506/) (2026)
- [On the scaling relationship between cloze probabilities and language model next-token prediction](https://mlsanity.com/paper/2602.17848/) (2026)
- [Beyond Local Surprise: Grounded Dialogue as Selective Belief Revision under Referential Uncertainty](https://mlsanity.com/paper/2608.26035/) (2026)
- [Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy](https://mlsanity.com/paper/2608.27813/) (2026)

## Abstract (arXiv)

Surprisal theory posits that the cognitive effort required to comprehend a word is determined by its contextual predictability, quantified as surprisal. Traditionally, surprisal theory treats words as distinct entities, overlooking any potential similarity between them. Giulianelli et al. (2023) address this limitation by introducing information value, a measure of predictability designed to account for similarities between communicative units. Our work leverages Ricotta and Szeidl's (2006) diversity index to extend surprisal into a metric that we term similarity-adjusted surprisal, exposing a mathematical relationship between surprisal and information value. Similarity-adjusted surprisal aligns with information value when considering graded similarities and reduces to standard surprisal when words are treated as distinct. Experimental results with reading time data indicate that similarity-adjusted surprisal adds predictive power beyond standard surprisal for certain datasets, suggesting it serves as a complementary measure of comprehension effort.