mlsanityData to 7 Oct 2026

Generalized Measures of Anticipation and Responsivity in Online Language Processing

arXiv 2409.10728 ↗First posted 16 Sep 2024cs.CL cs.AI cs.ITcited by 23

Not on the map. Why?

Abstract (arXiv)

We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations of incremental linguistic contexts. Our framework provides a formal definition of anticipatory and responsive measures, and it equips experimenters with the tools to define new, more expressive measures beyond standard next-symbol entropy and surprisal. While extracting these standard quantities from language models is convenient, we demonstrate that using Monte Carlo simulation to estimate alternative responsive and anticipatory measures pays off empirically: New special cases of our generalized formula exhibit enhanced predictive power compared to surprisal for human cloze completion probability as well as ELAN, LAN, and N400 amplitudes, and greater complementarity with surprisal in predicting reading times.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 9 of its 74 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

9 of 74 references.

References · 9

  1. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  2. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 75
  3. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
  4. 2017The Natural Stories Corpuscited by 40
  5. 2022Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguitiescited by 30
  6. 2023Information Value: Measuring Utterance Predictability as Distance from Plausible Alternativescited by 22
  7. 2022On the Effect of Anticipation on Reading Timescited by 18
  8. 2024On the Role of Context in Reading Time Predictioncited by 17
  9. 2022Towards Pragmatic Production Strategies for Natural Language Generation Taskscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 10.

23 papers.

Sort:

  1. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 50
  2. 2025Large Language Models Are Human-Like Internallycited by 28
  3. 2024Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discoursecited by 14
  4. 2024On the Proper Treatment of Tokenization in Psycholinguisticscited by 12
  5. 2024Towards a Similarity-adjusted Surprisal Theorycited by 9
  6. 2025The Harmonic Structure of Information Contourscited by 7
  7. 2024Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically alignedcited by 4
  8. 2025How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?cited by 3
  9. 2026Is Information Density Uniform when Utterances are Grounded on Perception and Discourse?cited by 3
  10. 2025A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behaviorcited by 2
  11. 2026Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisalcited by 1
  12. 2026Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialoguecited by 1
11 more
  1. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  2. 2025Large language models have learned to use languagecited by 0
  3. 2025Large language models are not about natural languagecited by 0
  4. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  5. 2026Are Language Models Models?cited by 0
  6. 2026On the scaling relationship between cloze probabilities and language model next-token predictioncited by 0
  7. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
  8. 2026You Can't Fight in Here! This is BBS!cited by 0
  9. 2026Probing for Reading Timescited by 0
  10. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0
  11. 2026Surprisal Theory is Tautological (without Rational Grounding)cited by 0