mlsanityData to 14 Oct 2026

Surprisal Theory is Tautological (without Rational Grounding)

arXiv 2607.21574 ↗First posted 23 Jul 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Surprisal theory holds that the human processing difficulty of a linguistic unit in context is an affine function of its surprisal under some language model. I argue this claim is a tautology without further constraint: for any non-negative difficulty measure over units in context, there exists a language model whose surprisal is an affine function of it under mild technical conditions. Therefore, because any pattern of difficulty is consistent with some language model, without an additional constraint on the language model, surprisal theory makes no falsifiable predictions. The tautology was long obscured by an assumption implicit in two decades of psycholinguistic work---that the relevant language model is the distribution that generated the training corpus, so that improving corpus fit improves predictions of human behavior. Recent empirical work has undermined this assumption, demonstrating that better corpus models can be worse predictors of processing difficulty. I conclude that breaking the tautology requires a rationalist intervention, i.e., the relevant language model must be derived from a non-empirically motivated model of the comprehender, which could be based on, for instance, memory constraints or processing goals, and that, thus, does not depend on the behavioral data surprisal theory is meant to explain.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 75 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

20 of 75 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
  3. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  4. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  5. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
  6. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  7. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  8. 2022A Measure-Theoretic Characterization of Tight Language Modelscited by 33
  9. 2025Large Language Models Are Human-Like Internallycited by 29
  10. 2024Generalized Measures of Anticipation and Responsivity in Online Language Processingcited by 24
  11. 2023A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processingcited by 22
  12. 2024On the Role of Context in Reading Time Predictioncited by 20
8 more
  1. 2025Characterizing the Expressivity of Fixed-Precision Transformer Language Modelscited by 14
  2. 2022Analyzing Wrap-Up Effects through an Information-Theoretic Lenscited by 10
  3. 2025The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakagecited by 5
  4. 2026Transducing Language Modelscited by 5
  5. 2025Probability Distributions Computed by Autoregressive Transformerscited by 2
  6. 2026On the Proper Treatment of Units in Surprisal Theorycited by 2
  7. 2025Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorlycited by 1
  8. 2026An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisalcited by 1

Built on by

Not cited yet.