mlsanityData to 7 Oct 2026

On the Proper Treatment of Units in Surprisal Theory

arXiv 2604.28147 ↗First posted 30 Apr 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units. As a result, surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices: the definition of the unit of analysis and the choice of regions of interest over which predictions are evaluated. In this paper, we disentangle these choices and give a unified framework for reasoning about surprisal over arbitrary unit inventories. We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail rather than a scientific primitive.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 98 references that are in the map. First posted on arXiv 30 Apr 2026.

Builds on

22 of 98 references.

References · 22

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  4. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  5. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 75
  6. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  7. 2021Revisiting the Uniform Information Density Hypothesiscited by 63
  8. 2024How to Compute the Probability of a Wordcited by 36
  9. 2022Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguitiescited by 30
  10. 2023Analyzing Cognitive Plausibility of Subword Tokenizationcited by 30
  11. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  12. 2024From Language Models over Tokens to Language Models over Characterscited by 28
10 more
  1. 2025Large Language Models Are Human-Like Internallycited by 28
  2. 2023Information Value: Measuring Utterance Predictability as Distance from Plausible Alternativescited by 22
  3. 2024On the Role of Context in Reading Time Predictioncited by 17
  4. 2024Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discoursecited by 14
  5. 2024The Effect of Surprisal on Reading Times in Information Seeking and Repeated Readingcited by 8
  6. 2025The Harmonic Structure of Information Contourscited by 7
  7. 2024Constructing a BPE Tokenization DFAcited by 5
  8. 2024The Impact of Token Granularity on the Predictive Power of Language Model Surprisalcited by 5
  9. 2026Transducing Language Modelscited by 3
  10. 2025A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behaviorcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Surprisal Theory is Tautological (without Rational Grounding)cited by 0
  2. 2026Stochastic Estimation of Transduced Language Modelscited by 0