# Generalized Measures of Anticipation and Responsivity in Online Language Processing

arXiv: https://arxiv.org/abs/2409.10728 · HTML: https://mlsanity.com/paper/2409.10728/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 74 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Builds on (most cited first, 9 of 9)

- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [Human Sentence Processing: Recurrence or Attention?](https://mlsanity.com/paper/2005.09471/) (2020)
- [The Natural Stories Corpus](https://mlsanity.com/paper/1708.05763/) (2017)
- [Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities](https://mlsanity.com/paper/2210.12187/) (2022)
- [Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives](https://mlsanity.com/paper/2310.13676/) (2023)
- [On the Role of Context in Reading Time Prediction](https://mlsanity.com/paper/2409.08160/) (2024)
- [On the Effect of Anticipation on Reading Times](https://mlsanity.com/paper/2211.14301/) (2022)
- [Towards Pragmatic Production Strategies for Natural Language Generation Tasks](https://mlsanity.com/paper/2210.12828/) (2022)

## Built on by (most cited first, 24 of 24)

- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Large Language Models Are Human-Like Internally](https://mlsanity.com/paper/2502.01615/) (2025)
- [Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse](https://mlsanity.com/paper/2410.16062/) (2024)
- [On the Proper Treatment of Tokenization in Psycholinguistics](https://mlsanity.com/paper/2410.02691/) (2024)
- [Towards a Similarity-adjusted Surprisal Theory](https://mlsanity.com/paper/2410.17676/) (2024)
- [The Harmonic Structure of Information Contours](https://mlsanity.com/paper/2506.03902/) (2025)
- [Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned](https://mlsanity.com/paper/2410.12057/) (2024)
- [How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?](https://mlsanity.com/paper/2507.22209/) (2025)
- [A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behavior](https://mlsanity.com/paper/2506.19999/) (2025)
- [Is Information Density Uniform when Utterances are Grounded on Perception and Discourse?](https://mlsanity.com/paper/2602.14653/) (2026)
- [Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal](https://mlsanity.com/paper/2601.09886/) (2026)
- [Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialogue](https://mlsanity.com/paper/2605.00506/) (2026)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)
- [Linguists should learn to love speech-based deep learning models](https://mlsanity.com/paper/2512.14506/) (2025)
- [Are Language Models Models?](https://mlsanity.com/paper/2601.10421/) (2026)
- [On the scaling relationship between cloze probabilities and language model next-token prediction](https://mlsanity.com/paper/2602.17848/) (2026)
- [Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities](https://mlsanity.com/paper/2604.09466/) (2026)
- [You Can't Fight in Here! This is BBS!](https://mlsanity.com/paper/2604.09501/) (2026)
- [Probing for Reading Times](https://mlsanity.com/paper/2604.18712/) (2026)
- [Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear](https://mlsanity.com/paper/2605.10061/) (2026)
- [Surprisal Theory is Tautological (without Rational Grounding)](https://mlsanity.com/paper/2607.21574/) (2026)
- [Revisiting the Estimation of Unigram Surprisal in Computational Psycholinguistics](https://mlsanity.com/paper/2610.18414/) (2026)

## Abstract (arXiv)

We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations of incremental linguistic contexts. Our framework provides a formal definition of anticipatory and responsive measures, and it equips experimenters with the tools to define new, more expressive measures beyond standard next-symbol entropy and surprisal. While extracting these standard quantities from language models is convenient, we demonstrate that using Monte Carlo simulation to estimate alternative responsive and anticipatory measures pays off empirically: New special cases of our generalized formula exhibit enhanced predictive power compared to surprisal for human cloze completion probability as well as ELAN, LAN, and N400 amplitudes, and greater complementarity with surprisal in predicting reading times.