mlsanityData to 14 Oct 2026

The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage

arXiv 2506.01172 ↗First posted 1 Jun 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

In psycholinguistic modeling, surprisal from larger pre-trained language models has been shown to be a poorer predictor of naturalistic human reading times. However, it has been speculated that this may be due to data leakage that caused language models to see the text stimuli during training. This paper presents two studies to address this concern at scale. The first study reveals relatively little leakage of five naturalistic reading time corpora in two pre-training datasets in terms of length and frequency of token n-gram overlap. The second study replicates the negative relationship between language model size and the fit of surprisal to reading times using models trained on 'leakage-free' data that overlaps only minimally with the reading time corpora. Taken together, this suggests that previous results using language models trained on these corpora are not driven by the effects of data leakage.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 13 of its 43 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

13 of 43 references.

References · 13

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  5. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 113
  6. 2020Syntactic Structure from Deep Learningcited by 99
  7. 2023Testing the Predictions of Surprisal Theory in 11 Languagescited by 77
  8. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 41
  9. 2017The Natural Stories Corpuscited by 40
  10. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 39
  11. 2024How to Compute the Probability of a Wordcited by 36
  12. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
1 more
  1. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026N-gram-like Language Models Predict Naturalistic Reading Time Bestcited by 2
  2. 2026On the scaling relationship between cloze probabilities and language model next-token predictioncited by 0
  3. 2026Surprisal Theory is Tautological (without Rational Grounding)cited by 0
  4. 2026A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinesecited by 0
  5. 2026Estimating Surprisal from the Speech Signalcited by 0